Statistics

Every crawler collects runtime statistics about the requests it processes. It counts finished and failed requests, measures how long they take, records how many retries each one needed, and groups the errors it hits. The Statistics class does the bookkeeping. The crawler calls it whenever a request starts, finishes, fails, or gets retried, so you don't have to record anything yourself.

What gets tracked

The raw data lives in a StatisticsState model. Its main fields are:

• requests_finished and requests_failed count requests that succeeded or failed after exhausting their retries.
• requests_retries counts requests that needed at least one retry, and request_retry_histogram shows how many requests finished or failed after 0, 1, 2, and more retries.
• request_total_finished_duration, request_total_failed_duration, request_min_duration, and request_max_duration measure request processing time.
• requests_with_status_code maps HTTP status codes to the number of responses that returned them.
• crawler_started_at, crawler_finished_at, and the computed crawler_runtime describe the run itself.

From the state, Statistics.calculate derives a FinalStatistics object. It's a frozen summary with the request counts, the retry histogram as a list, average finished and failed durations, total duration, per-minute rates for finished and failed requests, and the crawler runtime. The same summary appears in the periodic and final statistics logs.

Reading statistics

There are two ways to get the statistics of a crawl:

• crawler.run() returns the FinalStatistics of the run once it ends.
• crawler.statistics gives you the Statistics instance, so you can read the full state or call calculate() at any time, including from a request handler while the crawl is running.

The following example reads the summary returned by run(), then looks at state fields that FinalStatistics doesn't include, and the errors collected during the run.

Run on

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=10)

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        await context.enqueue_links()

    # `run()` returns a snapshot of the statistics at the end of the run.
    final_stats = await crawler.run(['https://crawlee.dev'])

    print(f'Finished requests: {final_stats.requests_finished}')
    print(f'Failed requests: {final_stats.requests_failed}')
    print(f'Average duration: {final_stats.request_avg_finished_duration}')
    print(f'Crawler runtime: {final_stats.crawler_runtime}')

    # The full state, including fields that `FinalStatistics` doesn't expose.
    state = crawler.statistics.state
    print(f'Status codes: {state.requests_with_status_code}')
    print(f'Slowest request: {state.request_max_duration}')

    # Errors grouped by similarity.
    error_tracker = crawler.statistics.error_tracker
    print(f'Total errors: {error_tracker.total}')
    print(f'Most common errors: {error_tracker.get_most_common_errors()}')


if __name__ == '__main__':
    asyncio.run(main())

A crawler that you run again with run() after a completed run resets its statistics at the start of the new run. To keep the numbers from a previous run, store the returned FinalStatistics before calling run() again.

Periodic logging

While the crawler runs, Statistics logs the current FinalStatistics at a fixed interval, and the crawler logs a final summary when the run ends. Two options control the periodic messages, and both belong to Statistics:

• log_interval sets how often the statistics are logged. The default is one minute.
• statistics_log_format picks between table, a formatted table, and inline, a single message with the values attached as extra log fields.

The crawler's constructor also accepts statistics_log_format. When the crawler creates its own Statistics, it passes the value through, so the option controls both the periodic and the final logs. When you pass a custom Statistics, the crawler's option applies only to the final summary. For more about log output, including structured JSON logs, see Logging.

Persistence

Statistics can save its state to a key-value store, so a crawl that gets migrated, aborted, or restarted continues counting from where it left off. When persistence is on, the state is loaded from the store when the crawl starts, saved on every PERSIST_STATE event, and saved once more when the crawl ends. For details about the event, see Events.

The following options control persistence:

• persistence_enabled turns persistence on. With 'explicit_only', a saved state is loaded at the start, but nothing is saved automatically, neither on PERSIST_STATE events nor at the end of the crawl.
• persist_state_key is the key the state is stored under. It defaults to __CRAWLER_STATISTICS_<n>, where <n> is a per-process counter of Statistics instances.
• persist_state_kvs_name selects a named key-value store. Without it, the default key-value store is used.

The statistics a crawler creates for itself have persistence enabled and use the crawler's key-value store. The exception is AdaptivePlaywrightCrawler, whose default statistics don't persist. A Statistics instance you create yourself has persistence_enabled=False by default. For how to resume an interrupted crawl, see Stopping and resuming crawlers.

Custom statistics

To change any of these options, create a Statistics instance and pass it to the crawler through the statistics argument. Use Statistics.with_default_state to get an instance with the standard StatisticsState model.

The following example logs the statistics every second in the inline format, persists them under a custom key, and reads the persisted record back after the run.

Run on

import asyncio
from datetime import timedelta

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.statistics import Statistics


async def main() -> None:
    statistics = Statistics.with_default_state(
        # Log the current statistics every second.
        log_interval=timedelta(seconds=1),
        log_message='Crawler statistics',
        # Log values as extra fields of a single message.
        statistics_log_format='inline',
        # Persist the statistics to the default key-value store.
        persistence_enabled=True,
        persist_state_key='MY_CRAWLER_STATISTICS',
    )

    crawler = ParselCrawler(
        statistics=statistics,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        await context.enqueue_links()

    await crawler.run(['https://crawlee.dev'])

    # Read the persisted statistics back from the key-value store.
    kvs = await crawler.get_key_value_store()
    persisted = await kvs.get_value('MY_CRAWLER_STATISTICS')
    print(f'Persisted finished requests: {persisted["requestsFinished"]}')


if __name__ == '__main__':
    asyncio.run(main())

A custom Statistics logs its periodic messages through the crawlee.statistics._statistics logger. To log through the crawler's logger instead, pass it as periodic_message_logger.

Error tracking

Each Statistics instance holds an error_tracker, which groups errors from failed request attempts by their type, the place in the code where they were raised, and a similar message. Its total and unique_error_count properties count all errors and the distinct groups, and get_most_common_errors() returns the most frequent groups with their counts. At the end of a run that had errors, the crawler logs both counts. A second tracker, error_tracker_retry, collects session errors that caused a retry with a new session.

With save_error_snapshots=True, the error tracker also saves a snapshot of the page on the first occurrence of each unique error. For details and examples, see Capturing page snapshots on errors.