ParsedHttpCrawlingContext

The crawling context used by AbstractHttpCrawler.

It provides access to key objects as well as utility functions for handling crawling tasks.

Hierarchy

• HttpCrawlingContext
  • ParsedHttpCrawlingContext
    • ParselCrawlingContext
    • BeautifulSoupCrawlingContext

Index

Methods

• from_basic_crawling_context
• from_http_crawling_context

Properties

• add_requests
• enqueue_links
• get_key_value_store
• http_response
• log
• parsed_content
• proxy_info
• push_data
• request
• send_request
• session
• use_state

Methods

from_basic_crawling_context

• from_basic_crawling_context(*, context, http_response): Self

• Convenience constructor that creates HttpCrawlingContext from existing BasicCrawlingContext.

  ---

  Parameters

  • optionalkeyword-onlycontext: BasicCrawlingContext

  • optionalkeyword-onlyhttp_response: HttpResponse

  Returns Self

from_http_crawling_context

• from_http_crawling_context(*, context, parsed_content, enqueue_links): Self

• Convenience constructor that creates new context from existing HttpCrawlingContext.

  ---

  Parameters

  • optionalkeyword-onlycontext: HttpCrawlingContext

  • optionalkeyword-onlyparsed_content: TParseResult

  • optionalkeyword-onlyenqueue_links: EnqueueLinksFunction

  Returns Self

Properties

add_requests

add_requests: AddRequestsFunction

Inherited from BasicCrawlingContext.add_requests

Add requests crawling context helper function.

enqueue_links

enqueue_links: EnqueueLinksFunction

get_key_value_store

get_key_value_store: GetKeyValueStoreFromRequestHandlerFunction

Inherited from BasicCrawlingContext.get_key_value_store

Get key-value store crawling context helper function.

http_response

http_response: HttpResponse

Inherited from HttpCrawlingResult.http_response

The HTTP response received from the server.

log

log: logging.Logger

Inherited from BasicCrawlingContext.log

Logger instance.

parsed_content

parsed_content: TParseResult

proxy_info

proxy_info: ProxyInfo | None

Inherited from BasicCrawlingContext.proxy_info

Proxy information for the current page being processed.

push_data

push_data: PushDataFunction

Inherited from BasicCrawlingContext.push_data

Push data crawling context helper function.

request

request: Request

Inherited from BasicCrawlingContext.request

Request object for the current page being processed.

send_request

send_request: SendRequestFunction

Inherited from BasicCrawlingContext.send_request

Send request crawling context helper function.

session

session: Session | None

Inherited from BasicCrawlingContext.session

Session object for the current page being processed.

use_state

use_state: UseStateFunction

Inherited from BasicCrawlingContext.use_state

Use state crawling context helper function.