ParselCrawler
Hierarchy
- BasicCrawler
- ParselCrawler
Index
Errors
error_handler
Decorator for configuring an error handler (called after a request handler error and before retrying).
Parameters
optionalkeyword-onlyhandler: ErrorHandler[TCrawlingContext | BasicCrawlingContext]
Returns ErrorHandler[TCrawlingContext]
Methods
__init__
A default constructor.
Parameters
optionalkeyword-onlyadditional_http_error_status_codes: Iterable[int] = ()
Additional HTTP status codes to treat as errors, triggering automatic retries when encountered.
optionalkeyword-onlyignore_http_error_status_codes: Iterable[int] = ()
HTTP status codes typically considered errors but to be treated as successful responses.
keyword-onlyrequest_provider: RequestProvider
Provider for requests to be processed by the crawler.
keyword-onlyrequest_handler: Callable[[TCrawlingContext], Awaitable[None]]
A callable responsible for handling requests.
keyword-onlyhttp_client: BaseHttpClient
HTTP client used by
BasicCrawlingContext.send_request
and the HTTP-based crawling.keyword-onlyconcurrency_settings: ConcurrencySettings
Settings to fine-tune concurrency levels.
keyword-onlymax_request_retries: int
Maximum number of attempts to process a single request.
keyword-onlymax_requests_per_crawl: int | None
Maximum number of pages to open during a crawl. The crawl stops upon reaching this limit. Setting this value can help avoid infinite loops in misconfigured crawlers.
None
means no limit. Due to concurrency settings, the actual number of pages visited may slightly exceed this value.keyword-onlymax_session_rotations: int
Maximum number of session rotations per request. The crawler rotates the session if a proxy error occurs or if the website blocks the request.
keyword-onlyconfiguration: Configuration
Crawler configuration.
keyword-onlyrequest_handler_timeout: timedelta
Maximum duration allowed for a single request handler to run.
keyword-onlyuse_session_pool: bool
Enable the use of a session pool for managing sessions during crawling.
keyword-onlysession_pool: SessionPool
A custom
SessionPool
instance, allowing the use of non-default configuration.keyword-onlyretry_on_blocked: bool
If True, the crawler attempts to bypass bot protections automatically.
keyword-onlyproxy_configuration: ProxyConfiguration
HTTP proxy configuration used when making requests.
keyword-onlystatistics: Statistics[StatisticsState]
A custom
Statistics
instance, allowing the use of non-default configuration.keyword-onlyevent_manager: EventManager
A custom
EventManager
instance, allowing the use of non-default configuration.keyword-onlyconfigure_logging: bool
If True, the crawler will set up logging infrastructure automatically.
keyword-onlymax_crawl_depth: int | None
Limits crawl depth from 0 (initial requests) up to the specified
max_crawl_depth
. Requests at the maximum depth are processed, but no further links are enqueued.keyword-onlyabort_on_error: bool
If True, the crawler stops immediately when any request handler error occurs.
Returns None
add_requests
Add requests to the underlying request provider in batches.
Parameters
optionalkeyword-onlyrequests: Sequence[str | Request]
A list of requests to add to the queue.
optionalkeyword-onlybatch_size: int = 1000
The number of requests to add in one batch.
optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(0)
Time to wait between adding batches.
optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
If True, wait for all requests to be added before returning.
optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None
Timeout for waiting for all requests to be added.
Returns None
export_data
Export data from a dataset.
This helper method simplifies the process of exporting data from a dataset. It opens the specified dataset and then exports the data based on the provided parameters. If you need to pass options specific to the output format, use the
export_data_csv
orexport_data_json
method instead.Parameters
optionalkeyword-onlypath: str | Path
The destination path.
optionalkeyword-onlydataset_id: str | None = None
The ID of the dataset.
optionalkeyword-onlydataset_name: str | None = None
The name of the dataset.
Returns None
export_data_csv
Export data from a dataset to a CSV file.
This helper method simplifies the process of exporting data from a dataset in csv format. It opens the specified dataset and then exports the data based on the provided parameters.
Parameters
optionalkeyword-onlypath: str | Path
The destination path.
optionalkeyword-onlydataset_id: str | None = None
The ID of the dataset.
optionalkeyword-onlydataset_name: str | None = None
The name of the dataset.
keyword-onlydialect: str
Specifies a dialect to be used in CSV parsing and writing.
keyword-onlydelimiter: str
A one-character string used to separate fields. Defaults to ','.
keyword-onlydoublequote: bool
Controls how instances of
quotechar
inside a field should be quoted. When True, the character is doubled; when False, theescapechar
is used as a prefix. Defaults to True.keyword-onlyescapechar: str
A one-character string used to escape the delimiter if
quoting
is set toQUOTE_NONE
and thequotechar
ifdoublequote
is False. Defaults to None, disabling escaping.keyword-onlylineterminator: str
The string used to terminate lines produced by the writer. Defaults to '\r\n'.
keyword-onlyquotechar: str
A one-character string used to quote fields containing special characters, like the delimiter or quotechar, or fields containing new-line characters. Defaults to '"'.
keyword-onlyquoting: int
Controls when quotes should be generated by the writer and recognized by the reader. Can take any of the
QUOTE_*
constants, with a default ofQUOTE_MINIMAL
.keyword-onlyskipinitialspace: bool
When True, spaces immediately following the delimiter are ignored. Defaults to False.
keyword-onlystrict: bool
When True, raises an exception on bad CSV input. Defaults to False.
Returns None
export_data_json
Export data from a dataset to a JSON file.
This helper method simplifies the process of exporting data from a dataset in json format. It opens the specified dataset and then exports the data based on the provided parameters.
Parameters
optionalkeyword-onlypath: str | Path
The destination path
optionalkeyword-onlydataset_id: str | None = None
The ID of the dataset.
optionalkeyword-onlydataset_name: str | None = None
The name of the dataset.
keyword-onlyskipkeys: bool
If True (default: False), dict keys that are not of a basic type (str, int, float, bool, None) will be skipped instead of raising a
TypeError
.keyword-onlyensure_ascii: bool
Determines if non-ASCII characters should be escaped in the output JSON string.
keyword-onlycheck_circular: bool
If False (default: True), skips the circular reference check for container types. A circular reference will result in a
RecursionError
or worse if unchecked.keyword-onlyallow_nan: bool
If False (default: True), raises a ValueError for out-of-range float values (nan, inf, -inf) to strictly comply with the JSON specification. If True, uses their JavaScript equivalents (NaN, Infinity, -Infinity).
keyword-onlycls: type[json.JSONEncoder]
Allows specifying a custom JSON encoder.
keyword-onlyindent: int
Specifies the number of spaces for indentation in the pretty-printed JSON output.
keyword-onlyseparators: tuple[str, str]
A tuple of (item_separator, key_separator). The default is (', ', ': ') if indent is None and (',', ': ') otherwise.
keyword-onlydefault: Callable
A function called for objects that can't be serialized otherwise. It should return a JSON-encodable version of the object or raise a
TypeError
.keyword-onlysort_keys: bool
Specifies whether the output JSON object should have keys sorted alphabetically.
Returns None
failed_request_handler
Decorator for configuring a failed request handler (called after max retries are reached).
Parameters
optionalkeyword-onlyhandler: FailedRequestHandler[TCrawlingContext | BasicCrawlingContext]
Returns FailedRequestHandler[TCrawlingContext]
get_data
Retrieve data from a dataset.
This helper method simplifies the process of retrieving data from a dataset. It opens the specified dataset and then retrieves the data based on the provided parameters.
Parameters
optionalkeyword-onlydataset_id: str | None = None
The ID of the dataset.
optionalkeyword-onlydataset_name: str | None = None
The name of the dataset.
keyword-onlyoffset: int
Skips the specified number of items at the start.
keyword-onlylimit: int
The maximum number of items to retrieve. Unlimited if None.
keyword-onlyclean: bool
Returns only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.
keyword-onlydesc: bool
Set to True to sort results in descending order.
keyword-onlyfields: list[str]
Fields to include in each item. Sorts fields as specified if provided.
keyword-onlyomit: list[str]
Fields to exclude from each item.
keyword-onlyunwind: str
Unwinds items by a specified array field, turning each element into a separate item.
keyword-onlyskip_empty: bool
Excludes empty items from the results if True.
keyword-onlyskip_hidden: bool
Excludes fields starting with '#' if True.
keyword-onlyflatten: list[str]
Fields to be flattened in returned items.
keyword-onlyview: str
Specifies the dataset view to be used.
Returns DatasetItemsListPage
get_dataset
Return the dataset with the given ID or name. If none is provided, return the default dataset.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
Returns Dataset
get_key_value_store
Return the key-value store with the given ID or name. If none is provided, return the default KVS.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
Returns KeyValueStore
get_request_provider
Return the configured request provider. If none is configured, open and return the default request queue.
Parameters
optionalkeyword-onlyid: str | None = None
optionalkeyword-onlyname: str | None = None
Returns RequestProvider
run
Run the crawler until all requests are processed.
Parameters
optionalkeyword-onlyrequests: Sequence[str | Request] | None = None
The requests to be enqueued before the crawler starts.
optionalkeyword-onlypurge_request_queue: bool = True
If this is
True
and the crawler is not being run for the first time, the default request queue will be purged.
Returns FinalStatistics
Properties
log
The logger used by the crawler.
router
The router used to handle each individual crawling request.
statistics
Statistics about the current (or last) crawler run.
A web crawler for performing HTTP requests and parsing HTML/XML content.
The
ParselCrawler
builds on top of theBasicCrawler
, which means it inherits all of its features. On top of that it implements the HTTP communication using the HTTP clients and HTML/XML parsing using theParsel
library. The class allows integration with any HTTP client that implements theBaseHttpClient
interface. The HTTP client is provided to the crawler as an input parameter to the constructor.The HTTP client-based crawlers are ideal for websites that do not require JavaScript execution. However, if you need to execute client-side JavaScript, consider using browser-based crawler like the
PlaywrightCrawler
.Usage