Skip to main content
Version: 1.11

ParselCrawlingContext

The crawling context used by the ParselCrawler.

It provides access to key objects as well as utility functions for handling crawling tasks.

Hierarchy

Index

Methods

Properties

Methods

__hash__

  • __hash__(): int

create_modified_copy

  • create_modified_copy(push_data, add_requests, get_key_value_store): Self

  • Create a modified copy of the crawling context with specified changes.

    Parameters

    • optionalpush_data: PushDataFunction | None = None
    • optionaladd_requests: AddRequestsFunction | None = None
    • optionalget_key_value_store: GetKeyValueStoreFromRequestHandlerFunction | None = None

    Returns Self

extract_form_requests

  • async extract_form_requests(*, selector, fields, click, all_forms, label, headers, session_id, keep_url_fragment, use_extended_unique_key, always_enqueue, user_data, no_retry, max_retries): list[Request]

  • Create requests submitting the forms matching selector the way a browser does.

    By default, only one form is submitted: the first one a browser can submit among those sharing the most field names with fields.

    Parameters

    • optionalkeyword-onlyselector: str = 'form'

      CSS selector for the forms. An invalid one matches no form.

    • optionalkeyword-onlyfields: Mapping[str, (str | Sequence[str]) | None] | None = None

      Values replacing the form fields or adding new ones. A None value drops the field and a sequence submits it once per value. With all_forms, only the fields a form has are replaced.

    • optionalkeyword-onlyclick: bool | Mapping[str, str] = True

      The submit button to click: True for the first enabled one, if any, False for none, or a mapping of attributes the button must have, disabled or not, which skips forms without such a button.

    • optionalkeyword-onlyall_forms: bool = False

      Submit each matching form instead of only one.

    • keyword-onlyoptionallabel: str | None

      A label routing the request to a specific handler.

    • keyword-onlyoptionalheaders: (HttpHeaders | dict[str, str]) | None

      HTTP headers of the request, replacing those the form sets, like Content-Type or Referer.

    • keyword-onlyoptionalsession_id: str | None

      ID of the Session the request is bound to.

    • keyword-onlyoptionalkeep_url_fragment: bool

      Whether the URL fragment counts towards the unique key of the request.

    • keyword-onlyoptionaluse_extended_unique_key: bool

      Whether the method and payload count towards the unique key. Defaults to True for POST forms.

    • keyword-onlyoptionalalways_enqueue: bool

      Whether to enqueue the request even if it's already in the queue.

    • keyword-onlyoptionaluser_data: Mapping[str, JsonSerializable]

      Custom data stored with the request.

    • keyword-onlyoptionalno_retry: bool

      Whether to skip retrying the request if it fails.

    • keyword-onlyoptionalmax_retries: int | None

      The maximum number of retries of the request.

    Returns list[Request]

from_basic_crawling_context

  • from_basic_crawling_context(context, http_response): Self

from_http_crawling_context

  • from_http_crawling_context(context, parsed_content, enqueue_links, extract_links): Self

from_parsed_http_crawling_context

  • from_parsed_http_crawling_context(context): Self

  • Create a new context from an existing ParsedHttpCrawlingContext[Selector].

    Parameters

    Returns Self

get_snapshot

html_to_text

  • html_to_text(): str

  • Convert the parsed HTML content to newline-separated plain text without tags.

    Returns str

Properties

add_requests

add_requests: AddRequestsFunction

Add requests crawling context helper function.

enqueue_links

enqueue_links: EnqueueLinksFunction

extract_links

extract_links: ExtractLinksFunction

get_key_value_store

get_key_value_store: GetKeyValueStoreFromRequestHandlerFunction

Get key-value store crawling context helper function.

http_response

http_response: HttpResponse

The HTTP response received from the server.

log

log: logging.Logger

Logger instance.

parsed_content

parsed_content: TParseResult

proxy_info

proxy_info: ProxyInfo | None

Proxy information for the current page being processed.

push_data

push_data: PushDataFunction

Push data crawling context helper function.

register_deferred_cleanup

register_deferred_cleanup: Callable[[DeferredCleanupCallback], None]

Register an async callback to be called after request processing completes (including error handlers).

request

request: Request

Request object for the current page being processed.

selector

selector: Selector

Convenience alias.

send_request

send_request: SendRequestFunction

Send request crawling context helper function.

session

session: Session | None

Session object for the current page being processed.

use_state

use_state: UseStateFunction

Use state crawling context helper function.

Page Options