PlaywrightCrawlerOptions
Hierarchy
- BasicCrawlerOptions
- _PlaywrightCrawlerAdditionalOptions
- PlaywrightCrawlerOptions
Index
Properties
- abort_on_error
- additional_http_error_status_codes
- browser_launch_options
- browser_new_context_options
- browser_pool
- browser_type
- concurrency_settings
- configuration
- configure_logging
- event_manager
- headless
- http_client
- ignore_http_error_status_codes
- keep_alive
- max_crawl_depth
- max_request_retries
- max_requests_per_crawl
- max_session_rotations
- proxy_configuration
- request_handler
- request_handler_timeout
- request_manager
- retry_on_blocked
- session_pool
- statistics
- storage_client
- use_session_pool
Properties
abort_on_error
If True, the crawler stops immediately when any request handler error occurs.
additional_http_error_status_codes
Additional HTTP status codes to treat as errors, triggering automatic retries when encountered.
browser_launch_options
Keyword arguments to pass to the browser launch method. These options are provided
directly to Playwright's browser_type.launch method. For more details, refer to the Playwright
documentation: https://playwright.dev/python/docs/api/class-browsertype#browser-type-launch.
This option should not be used if browser_pool is provided.
browser_new_context_options
Keyword arguments to pass to the browser new context method. These options are provided directly to Playwright's
browser.new_context method. For more details, refer to the Playwright documentation:
https://playwright.dev/python/docs/api/class-browser#browser-new-context. This option should not be used if
browser_pool is provided.
browser_pool
A BrowserPool instance to be used for launching the browsers and getting pages.
browser_type
The type of browser to launch ('chromium', 'firefox', or 'webkit').
This option should not be used if browser_pool is provided.
concurrency_settings
Settings to fine-tune concurrency levels.
configuration
The Configuration instance. Some of its properties are used as defaults for the crawler.
configure_logging
If True, the crawler will set up logging infrastructure automatically.
event_manager
The event manager for managing events for the crawler and all its components.
headless
Whether to run the browser in headless mode. This option should not be used if browser_pool is provided.
http_client
HTTP client used by BasicCrawlingContext.send_request method.
ignore_http_error_status_codes
HTTP status codes that are typically considered errors but should be treated as successful responses.
keep_alive
Flag that can keep crawler running even when there are no requests in queue.
max_crawl_depth
Specifies the maximum crawl depth. If set, the crawler will stop processing links beyond this depth. The crawl depth starts at 0 for initial requests and increases with each subsequent level of links. Requests at the maximum depth will still be processed, but no new links will be enqueued from those requests. If not set, crawling continues without depth restrictions.
max_request_retries
Maximum number of attempts to process a single request.
max_requests_per_crawl
Maximum number of pages to open during a crawl. The crawl stops upon reaching this limit.
Setting this value can help avoid infinite loops in misconfigured crawlers. None means no limit.
Due to concurrency settings, the actual number of pages visited may slightly exceed this value.
max_session_rotations
Maximum number of session rotations per request. The crawler rotates the session if a proxy error occurs or if the website blocks the request.
proxy_configuration
HTTP proxy configuration used when making requests.
request_handler
A callable responsible for handling requests.
request_handler_timeout
Maximum duration allowed for a single request handler to run.
request_manager
Manager of requests that should be processed by the crawler.
retry_on_blocked
If True, the crawler attempts to bypass bot protections automatically.
session_pool
A custom SessionPool instance, allowing the use of non-default configuration.
statistics
A custom Statistics instance, allowing the use of non-default configuration.
storage_client
The storage client for managing storages for the crawler and all its components.
use_session_pool
Enable the use of a session pool for managing sessions during crawling.
Arguments for the
AbstractHttpCrawlerconstructor.It is intended for typing forwarded
__init__arguments in the subclasses.