Skip to main content

Statistics

A class for collecting, tracking, and logging runtime statistics for requests.

It is designed to record information such as request durations, retries, successes, and failures, enabling analysis of crawler performance. The collected statistics are persisted to a KeyValueStore, ensuring they remain available across crawler migrations, abortions, and restarts. This persistence allows for tracking and evaluation of crawler behavior over its lifecycle.

Index

Methods

Properties

Methods

__aenter__

  • async __aenter__(): Self

  • Subscribe to events and start collecting statistics.

    Returns Self

__aexit__

  • async __aexit__(*, exc_type, exc_value, exc_traceback): None

  • Stop collecting statistics.

    Parameters

    • optionalkeyword-onlyexc_type: type[BaseException] | None
    • optionalkeyword-onlyexc_value: BaseException | None
    • optionalkeyword-onlyexc_traceback: TracebackType | None

    Returns None

__init__

  • __init__(*, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval, state_model): None

  • Parameters

    • optionalkeyword-onlypersistence_enabled: bool = False
    • optionalkeyword-onlypersist_state_kvs_name: str = 'default'
    • optionalkeyword-onlypersist_state_key: str | None = None
    • optionalkeyword-onlykey_value_store: KeyValueStore | None = None
    • optionalkeyword-onlylog_message: str = 'Statistics'
    • optionalkeyword-onlyperiodic_message_logger: Logger | None = None
    • optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)
    • optionalkeyword-onlystate_model: type[TStatisticsState] = cast(Any, StatisticsState)

    Returns None

calculate

record_request_processing_failure

  • record_request_processing_failure(*, request_id_or_key): None

  • Mark a request as failed.

    Parameters

    • optionalkeyword-onlyrequest_id_or_key: str

    Returns None

record_request_processing_finish

  • record_request_processing_finish(*, request_id_or_key): None

  • Mark a request as finished.

    Parameters

    • optionalkeyword-onlyrequest_id_or_key: str

    Returns None

record_request_processing_start

  • record_request_processing_start(*, request_id_or_key): None

  • Mark a request as started.

    Parameters

    • optionalkeyword-onlyrequest_id_or_key: str

    Returns None

register_status_code

  • register_status_code(*, code): None

  • Increment the number of times a status code has been received.

    Parameters

    • optionalkeyword-onlycode: int

    Returns None

reset

  • async reset(): None

  • Reset the statistics to their defaults and remove any persistent state.

    Returns None

Properties

active

active: bool

Indicate whether the context is active.

Page Options