BasicCrawlerOptions
Index
Properties
concurrency_settings
Settings to fine-tune concurrency levels.
configuration
Crawler configuration.
configure_logging
If True, the crawler will set up logging infrastructure automatically.
event_manager
A custom EventManager instance, allowing the use of non-default configuration.
http_client
HTTP client used by BasicCrawlingContext.send_request and the HTTP-based crawling.
max_crawl_depth
Limits crawl depth from 0 (initial requests) up to the specified max_crawl_depth.
Requests at the maximum depth are processed, but no further links are enqueued.
max_request_retries
Maximum number of attempts to process a single request.
max_requests_per_crawl
Maximum number of pages to open during a crawl. The crawl stops upon reaching this limit.
Setting this value can help avoid infinite loops in misconfigured crawlers. None means no limit.
Due to concurrency settings, the actual number of pages visited may slightly exceed this value.
max_session_rotations
Maximum number of session rotations per request. The crawler rotates the session if a proxy error occurs or if the website blocks the request.
proxy_configuration
HTTP proxy configuration used when making requests.
request_handler
A callable responsible for handling requests.
request_handler_timeout
Maximum duration allowed for a single request handler to run.
request_provider
Provider for requests to be processed by the crawler.
retry_on_blocked
If True, the crawler attempts to bypass bot protections automatically.
session_pool
A custom SessionPool instance, allowing the use of non-default configuration.
statistics
A custom Statistics instance, allowing the use of non-default configuration.
use_session_pool
Enable the use of a session pool for managing sessions during crawling.
Arguments for the
BasicCrawlerconstructor.It is intended for typing forwarded
__init__arguments in the subclasses.