BasicCrawlingContext

Basic crawling context.

It represents the fundamental crawling context used by the BasicCrawler. It is extended by more specific crawlers to provide additional functionality.

Hierarchy

• BasicCrawlingContext
  • AdaptivePlaywrightPreNavCrawlingContext
  • HttpCrawlingContext
  • PlaywrightPreNavCrawlingContext

Index

Methods

• __hash__
• create_modified_copy
• get_snapshot

Properties

• add_requests
• get_key_value_store
• log
• proxy_info
• push_data
• request
• send_request
• session
• use_state

Methods

__hash__

• __hash__(): int

• Return hash of the context. Each context is considered unique.

  ---

  Returns int

create_modified_copy

• create_modified_copy(push_data, add_requests, get_key_value_store): Self

• Create a modified copy of the crawling context with specified changes.

  ---

  Parameters

  • optionalpush_data: PushDataFunction | None = None

  • optionaladd_requests: AddRequestsFunction | None = None

  • optionalget_key_value_store: GetKeyValueStoreFromRequestHandlerFunction | None = None

  Returns Self

get_snapshot

• async get_snapshot(): PageSnapshot

• Get snapshot of crawled page.

  ---

  Returns PageSnapshot

Properties

add_requests

add_requests: AddRequestsFunction

Add requests crawling context helper function.

get_key_value_store

get_key_value_store: GetKeyValueStoreFromRequestHandlerFunction

Get key-value store crawling context helper function.

log

log: logging.Logger

Logger instance.

proxy_info

proxy_info: ProxyInfo | None

Proxy information for the current page being processed.

push_data

push_data: PushDataFunction

Push data crawling context helper function.

request

request: Request

Request object for the current page being processed.

send_request

send_request: SendRequestFunction

Send request crawling context helper function.

session

session: Session | None

Session object for the current page being processed.

use_state

use_state: UseStateFunction

Use state crawling context helper function.