ParselCrawler

A web crawler for performing HTTP requests and parsing HTML/XML content.

The ParselCrawler builds on top of the AbstractHttpCrawler, which means it inherits all of its features. It specifies its own parser ParselParser which is used to parse HttpResponse. ParselParser uses following library for parsing: https://pypi.org/project/parsel/

The HTTP client-based crawlers are ideal for websites that do not require JavaScript execution. However, if you need to execute client-side JavaScript, consider using browser-based crawler like the PlaywrightCrawler.

Usage

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext

crawler = ParselCrawler()

# Define the default request handler, which will be called for every request.
@crawler.router.default_handler
async def request_handler(context: ParselCrawlingContext) -> None:
    context.log.info(f'Processing {context.request.url} ...')

    # Extract data from the page.
    data = {
        'url': context.request.url,
        'title': context.selector.css('title').get(),
    }

    # Push the extracted data to the default dataset.
    await context.push_data(data)

await crawler.run(['https://crawlee.dev/'])

Hierarchy

AbstractHttpCrawler
- ParselCrawler

Methods

init

__init__(*, request_handler, statistics, configuration, event_manager, storage_client, request_manager, session_pool, proxy_configuration, http_client, max_request_retries, max_requests_per_crawl, max_session_rotations, max_crawl_depth, use_session_pool, retry_on_blocked, concurrency_settings, request_handler_timeout, abort_on_error, configure_logging, statistics_log_format, keep_alive, additional_http_error_status_codes, ignore_http_error_status_codes, respect_robots_txt_file): None

Overrides AbstractHttpCrawler.__init__
Initialize a new instance.
Parameters
- keyword-onlyoptionalrequest_handler: Callable[[TCrawlingContext], Awaitable[None]]
  A callable responsible for handling requests.
- keyword-onlyoptionalstatistics: Statistics[TStatisticsState]
  A custom Statistics instance, allowing the use of non-default configuration.
- keyword-onlyoptionalconfiguration: Configuration
  The Configuration instance. Some of its properties are used as defaults for the crawler.
- keyword-onlyoptionalevent_manager: EventManager
  The event manager for managing events for the crawler and all its components.
- keyword-onlyoptionalstorage_client: StorageClient
  The storage client for managing storages for the crawler and all its components.
- keyword-onlyoptionalrequest_manager: RequestManager
  Manager of requests that should be processed by the crawler.
- keyword-onlyoptionalsession_pool: SessionPool
  A custom SessionPool instance, allowing the use of non-default configuration.
- keyword-onlyoptionalproxy_configuration: ProxyConfiguration
  HTTP proxy configuration used when making requests.
- keyword-onlyoptionalhttp_client: HttpClient
  HTTP client used by BasicCrawlingContext.send_request method.
- keyword-onlyoptionalmax_request_retries: int
  Specifies the maximum number of retries allowed for a request if its processing fails. This includes retries due to navigation errors or errors thrown from user-supplied functions (request_handler, pre_navigation_hooks etc.).
  
  This limit does not apply to retries triggered by session rotation (see max_session_rotations).
- keyword-onlyoptionalmax_requests_per_crawl: int | None
  Maximum number of pages to open during a crawl. The crawl stops upon reaching this limit. Setting this value can help avoid infinite loops in misconfigured crawlers. None means no limit. Due to concurrency settings, the actual number of pages visited may slightly exceed this value.
- keyword-onlyoptionalmax_session_rotations: int
  Maximum number of session rotations per request. The crawler rotates the session if a proxy error occurs or if the website blocks the request.
  
  The session rotations are not counted towards the max_request_retries limit.
- keyword-onlyoptionalmax_crawl_depth: int | None
  Specifies the maximum crawl depth. If set, the crawler will stop processing links beyond this depth. The crawl depth starts at 0 for initial requests and increases with each subsequent level of links. Requests at the maximum depth will still be processed, but no new links will be enqueued from those requests. If not set, crawling continues without depth restrictions.
- keyword-onlyoptionaluse_session_pool: bool
  Enable the use of a session pool for managing sessions during crawling.
- keyword-onlyoptionalretry_on_blocked: bool
  If True, the crawler attempts to bypass bot protections automatically.
- keyword-onlyoptionalconcurrency_settings: ConcurrencySettings
  Settings to fine-tune concurrency levels.
- keyword-onlyoptionalrequest_handler_timeout: timedelta
  Maximum duration allowed for a single request handler to run.
- keyword-onlyoptionalabort_on_error: bool
  If True, the crawler stops immediately when any request handler error occurs.
- keyword-onlyoptionalconfigure_logging: bool
  If True, the crawler will set up logging infrastructure automatically.
- keyword-onlyoptionalstatistics_log_format: Literal[table, inline]
  If 'table', displays crawler statistics as formatted tables in logs. If 'inline', outputs statistics as plain text log messages.
- keyword-onlyoptionalkeep_alive: bool
  Flag that can keep crawler running even when there are no requests in queue.
- keyword-onlyoptionaladditional_http_error_status_codes: Iterable[int]
  Additional HTTP status codes to treat as errors, triggering automatic retries when encountered.
- keyword-onlyoptionalignore_http_error_status_codes: Iterable[int]
  HTTP status codes that are typically considered errors but should be treated as successful responses.
- keyword-onlyoptionalrespect_robots_txt_file: bool
  If set to True, the crawler will automatically try to fetch the robots.txt file for each domain, and skip those that are not allowed. This also prevents disallowed URLs to be added via EnqueueLinksFunction.
Returns None

add_requests

async add_requests(requests, *, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None

Inherited from BasicCrawler.add_requests
Add requests to the underlying request manager in batches.
Parameters
- requests: Sequence[str | Request]
  A list of requests to add to the queue.
- optionalkeyword-onlybatch_size: int = 1000
  The number of requests to add in one batch.
- optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(0)
  Time to wait between adding batches.
- optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
  If True, wait for all requests to be added before returning.
- optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None
  Timeout for waiting for all requests to be added.
Returns None

create_parsed_http_crawler_class

create_parsed_http_crawler_class(static_parser): type[AbstractHttpCrawler[ParsedHttpCrawlingContext[TParseResult], TParseResult, TSelectResult]]

Inherited from AbstractHttpCrawler.create_parsed_http_crawler_class
Create a specific version of AbstractHttpCrawler class.

This is a convenience factory method for creating a specific AbstractHttpCrawler subclass. While AbstractHttpCrawler allows its two generic parameters to be independent, this method simplifies cases where TParseResult is used for both generic parameters.
Parameters
- static_parser: AbstractHttpParser[TParseResult, TSelectResult]
Returns type[AbstractHttpCrawler[ParsedHttpCrawlingContext[TParseResult], TParseResult, TSelectResult]]

error_handler

error_handler(handler): ErrorHandler[TCrawlingContext]

Inherited from BasicCrawler.error_handler
Register a function to handle errors occurring in request handlers.

The error handler is invoked after a request handler error occurs and before a retry attempt.
Parameters
- handler: ErrorHandler[TCrawlingContext | BasicCrawlingContext]
Returns ErrorHandler[TCrawlingContext]

export_data

async export_data(path, dataset_id, dataset_name): None

Inherited from BasicCrawler.export_data
Export data from a Dataset.

This helper method simplifies the process of exporting data from a Dataset. It opens the specified one and then exports the data based on the provided parameters. If you need to pass options specific to the output format, use the export_data_csv or export_data_json method instead.
Parameters
- path: str | Path
  The destination path.
- optionaldataset_id: str | None = None
  The ID of the Dataset.
- optionaldataset_name: str | None = None
  The name of the Dataset.
Returns None

export_data_csv

async export_data_csv(path, *, dataset_id, dataset_name, dialect, delimiter, doublequote, escapechar, lineterminator, quotechar, quoting, skipinitialspace, strict): None

Inherited from BasicCrawler.export_data_csv
Export data from a Dataset to a CSV file.

This helper method simplifies the process of exporting data from a Dataset in csv format. It opens the specified one and then exports the data based on the provided parameters.
Parameters
- path: str | Path
  The destination path.
- optionalkeyword-onlydataset_id: str | None = None
  The ID of the Dataset.
- optionalkeyword-onlydataset_name: str | None = None
  The name of the Dataset.
- keyword-onlyoptionaldialect: str
  Specifies a dialect to be used in CSV parsing and writing.
- keyword-onlyoptionaldelimiter: str
  A one-character string used to separate fields. Defaults to ','.
- keyword-onlyoptionaldoublequote: bool
  Controls how instances of quotechar inside a field should be quoted. When True, the character is doubled; when False, the escapechar is used as a prefix. Defaults to True.
- keyword-onlyoptionalescapechar: str
  A one-character string used to escape the delimiter if quoting is set to QUOTE_NONE and the quotechar if doublequote is False. Defaults to None, disabling escaping.
- keyword-onlyoptionallineterminator: str
  The string used to terminate lines produced by the writer. Defaults to '\r\n'.
- keyword-onlyoptionalquotechar: str
  A one-character string used to quote fields containing special characters, like the delimiter or quotechar, or fields containing new-line characters. Defaults to '"'.
- keyword-onlyoptionalquoting: int
  Controls when quotes should be generated by the writer and recognized by the reader. Can take any of the QUOTE_* constants, with a default of QUOTE_MINIMAL.
- keyword-onlyoptionalskipinitialspace: bool
  When True, spaces immediately following the delimiter are ignored. Defaults to False.
- keyword-onlyoptionalstrict: bool
  When True, raises an exception on bad CSV input. Defaults to False.
Returns None

export_data_json

async export_data_json(path, *, dataset_id, dataset_name, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys): None

Inherited from BasicCrawler.export_data_json
Export data from a Dataset to a JSON file.

This helper method simplifies the process of exporting data from a Dataset in json format. It opens the specified one and then exports the data based on the provided parameters.
Parameters
- path: str | Path
  The destination path
- optionalkeyword-onlydataset_id: str | None = None
  The ID of the Dataset.
- optionalkeyword-onlydataset_name: str | None = None
  The name of the Dataset.
- keyword-onlyoptionalskipkeys: bool
  If True (default: False), dict keys that are not of a basic type (str, int, float, bool, None) will be skipped instead of raising a TypeError.
- keyword-onlyoptionalensure_ascii: bool
  Determines if non-ASCII characters should be escaped in the output JSON string.
- keyword-onlyoptionalcheck_circular: bool
  If False (default: True), skips the circular reference check for container types. A circular reference will result in a RecursionError or worse if unchecked.
- keyword-onlyoptionalallow_nan: bool
  If False (default: True), raises a ValueError for out-of-range float values (nan, inf, -inf) to strictly comply with the JSON specification. If True, uses their JavaScript equivalents (NaN, Infinity, -Infinity).
- keyword-onlyoptionalcls: type[json.JSONEncoder]
  Allows specifying a custom JSON encoder.
- keyword-onlyoptionalindent: int
  Specifies the number of spaces for indentation in the pretty-printed JSON output.
- keyword-onlyoptionalseparators: tuple[str, str]
  A tuple of (item_separator, key_separator). The default is (', ', ': ') if indent is None and (',', ': ') otherwise.
- keyword-onlyoptionaldefault: Callable
  A function called for objects that can't be serialized otherwise. It should return a JSON-encodable version of the object or raise a TypeError.
- keyword-onlyoptionalsort_keys: bool
  Specifies whether the output JSON object should have keys sorted alphabetically.
Returns None

failed_request_handler

failed_request_handler(handler): FailedRequestHandler[TCrawlingContext]

Inherited from BasicCrawler.failed_request_handler
Register a function to handle requests that exceed the maximum retry limit.

The failed request handler is invoked when a request has failed all retry attempts.
Parameters
- handler: FailedRequestHandler[TCrawlingContext | BasicCrawlingContext]
Returns FailedRequestHandler[TCrawlingContext]

get_data

async get_data(dataset_id, dataset_name, *, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): DatasetItemsListPage

Inherited from BasicCrawler.get_data
Retrieve data from a Dataset.

This helper method simplifies the process of retrieving data from a Dataset. It opens the specified one and then retrieves the data based on the provided parameters.
Parameters
- optionaldataset_id: str | None = None
  The ID of the Dataset.
- optionaldataset_name: str | None = None
  The name of the Dataset.
- keyword-onlyoptionaloffset: int
  Skip the specified number of items at the start.
- keyword-onlyoptionallimit: int
  The maximum number of items to retrieve. Unlimited if None.
- keyword-onlyoptionalclean: bool
  Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.
- keyword-onlyoptionaldesc: bool
  Set to True to sort results in descending order.
- keyword-onlyoptionalfields: list[str]
  Fields to include in each item. Sorts fields as specified if provided.
- keyword-onlyoptionalomit: list[str]
  Fields to exclude from each item.
- keyword-onlyoptionalunwind: str
  Unwind items by a specified array field, turning each element into a separate item.
- keyword-onlyoptionalskip_empty: bool
  Exclude empty items from the results if True.
- keyword-onlyoptionalskip_hidden: bool
  Exclude fields starting with '#' if True.
- keyword-onlyoptionalflatten: list[str]
  Field to be flattened in returned items.
- keyword-onlyoptionalview: str
  Specify the dataset view to be used.
Returns DatasetItemsListPage

get_dataset

async get_dataset(*, id, name): Dataset

Inherited from BasicCrawler.get_dataset
Return the Dataset with the given ID or name. If none is provided, return the default one.
Parameters
- optionalkeyword-onlyid: str | None = None
- optionalkeyword-onlyname: str | None = None
Returns Dataset

get_key_value_store

async get_key_value_store(*, id, name): KeyValueStore

Inherited from BasicCrawler.get_key_value_store
Return the KeyValueStore with the given ID or name. If none is provided, return the default KVS.
Parameters
- optionalkeyword-onlyid: str | None = None
- optionalkeyword-onlyname: str | None = None
Returns KeyValueStore

get_request_manager

async get_request_manager(): RequestManager

Inherited from BasicCrawler.get_request_manager
Return the configured request manager. If none is configured, open and return the default request queue.
Returns RequestManager

on_skipped_request

on_skipped_request(callback): SkippedRequestCallback

Inherited from BasicCrawler.on_skipped_request
Register a function to handle skipped requests.

The skipped request handler is invoked when a request is skipped due to a collision or other reasons.
Parameters
- callback: SkippedRequestCallback
Returns SkippedRequestCallback

pre_navigation_hook

pre_navigation_hook(hook): None

Inherited from AbstractHttpCrawler.pre_navigation_hook
Register a hook to be called before each navigation.
Parameters
- hook: Callable[[BasicCrawlingContext], Awaitable[None]]
  A coroutine function to be called before each navigation.
Returns None

run

async run(requests, *, purge_request_queue): FinalStatistics

Inherited from BasicCrawler.run
Run the crawler until all requests are processed.
Parameters
- optionalrequests: Sequence[str | Request] | None = None
  The requests to be enqueued before the crawler starts.
- optionalkeyword-onlypurge_request_queue: bool = True
  If this is True and the crawler is not being run for the first time, the default request queue will be purged.
Returns FinalStatistics

stop

stop(reason): None

Inherited from BasicCrawler.stop
Set flag to stop crawler.

This stops current crawler run regardless of whether all requests were finished.
Parameters
- optionalreason: str = 'Stop was called externally.'
  Reason for stopping that will be used in logs.
Returns None

Properties

log

log: logging.Logger

The logger used by the crawler.

router

router: Router[TCrawlingContext]

The Router used to handle each individual crawling request.

statistics

statistics: Statistics[TStatisticsState]

Statistics about the current (or last) crawler run.

Usage

Hierarchy

Index

Methods

Properties

Methods

__init__

Parameters

keyword-onlyoptionalrequest_handler: Callable[[TCrawlingContext], Awaitable[None]]

keyword-onlyoptionalstatistics: Statistics[TStatisticsState]

keyword-onlyoptionalconfiguration: Configuration

keyword-onlyoptionalevent_manager: EventManager

keyword-onlyoptionalstorage_client: StorageClient

keyword-onlyoptionalrequest_manager: RequestManager

keyword-onlyoptionalsession_pool: SessionPool

keyword-onlyoptionalproxy_configuration: ProxyConfiguration

keyword-onlyoptionalhttp_client: HttpClient

keyword-onlyoptionalmax_request_retries: int

keyword-onlyoptionalmax_requests_per_crawl: int | None

keyword-onlyoptionalmax_session_rotations: int

keyword-onlyoptionalmax_crawl_depth: int | None

keyword-onlyoptionaluse_session_pool: bool

keyword-onlyoptionalretry_on_blocked: bool

keyword-onlyoptionalconcurrency_settings: ConcurrencySettings

keyword-onlyoptionalrequest_handler_timeout: timedelta

keyword-onlyoptionalabort_on_error: bool

keyword-onlyoptionalconfigure_logging: bool

keyword-onlyoptionalstatistics_log_format: Literal[table, inline]

keyword-onlyoptionalkeep_alive: bool

keyword-onlyoptionaladditional_http_error_status_codes: Iterable[int]

keyword-onlyoptionalignore_http_error_status_codes: Iterable[int]

keyword-onlyoptionalrespect_robots_txt_file: bool

Returns None

add_requests

Parameters

requests: Sequence[str | Request]

optionalkeyword-onlybatch_size: int = 1000

optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(0)

optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False

optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None

Returns None

create_parsed_http_crawler_class

Parameters

static_parser: AbstractHttpParser[TParseResult, TSelectResult]

Returns type[AbstractHttpCrawler[ParsedHttpCrawlingContext[TParseResult], TParseResult, TSelectResult]]

error_handler

Parameters

handler: ErrorHandler[TCrawlingContext | BasicCrawlingContext]

Returns ErrorHandler[TCrawlingContext]

export_data

Parameters

path: str | Path

optionaldataset_id: str | None = None

optionaldataset_name: str | None = None

Returns None

export_data_csv

Parameters

path: str | Path

optionalkeyword-onlydataset_id: str | None = None

optionalkeyword-onlydataset_name: str | None = None

keyword-onlyoptionaldialect: str

keyword-onlyoptionaldelimiter: str

keyword-onlyoptionaldoublequote: bool

keyword-onlyoptionalescapechar: str

keyword-onlyoptionallineterminator: str

keyword-onlyoptionalquotechar: str

keyword-onlyoptionalquoting: int

keyword-onlyoptionalskipinitialspace: bool

keyword-onlyoptionalstrict: bool

Returns None

export_data_json

Parameters

path: str | Path

optionalkeyword-onlydataset_id: str | None = None

optionalkeyword-onlydataset_name: str | None = None

keyword-onlyoptionalskipkeys: bool

keyword-onlyoptionalensure_ascii: bool

keyword-onlyoptionalcheck_circular: bool

keyword-onlyoptionalallow_nan: bool

keyword-onlyoptionalcls: type[json.JSONEncoder]

init