Skip to main content

ParsedHttpCrawlingContext

The crawling context used by AbstractHttpCrawler.

It provides access to key objects as well as utility functions for handling crawling tasks.

Hierarchy

Index

Methods

Properties

Methods

from_basic_crawling_context

  • from_basic_crawling_context(*, context, http_response): Self

  • Convenience constructor that creates HttpCrawlingContext from existing BasicCrawlingContext.

    Parameters

    Returns Self

from_http_crawling_context

  • from_http_crawling_context(*, context, parsed_content, enqueue_links): Self

  • Convenience constructor that creates new context from existing HttpCrawlingContext.

    Parameters

    Returns Self

Properties

add_requests

add_requests: AddRequestsFunction

enqueue_links

enqueue_links: EnqueueLinksFunction

get_key_value_store

get_key_value_store: GetKeyValueStoreFromRequestHandlerFunction

http_response

http_response: HttpResponse

The HTTP response received from the server.

log

log: logging.Logger

parsed_content

parsed_content: TParseResult

proxy_info

proxy_info: ProxyInfo | None

push_data

push_data: PushDataFunction

request

request: Request

send_request

send_request: SendRequestFunction

session

session: Session | None

use_state

use_state: UseStateFunction
Page Options