Skip to main content

Statistics

crawlee.statistics._statistics.Statistics

An interface to collecting and logging runtime statistics for requests.

All information is saved to the key value store so that it persists between migrations, abortions and resurrections.

Index

Constructors

Methods

Constructors

__init__

  • __init__(*, event_manager, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval, state_model): None

  • Parameters

    • event_manager: EventManager | None = Nonekeyword-only
    • persistence_enabled: bool = Falsekeyword-only
    • persist_state_kvs_name: str = 'default'keyword-only
    • persist_state_key: str | None = Nonekeyword-only
    • key_value_store: KeyValueStore | None = Nonekeyword-only
    • log_message: str = 'Statistics'keyword-only
    • periodic_message_logger: Logger | None = Nonekeyword-only
    • log_interval: timedelta = timedelta(minutes=1)keyword-only
    • state_model: type[TStatisticsState] = cast(Any, StatisticsState)keyword-only

    Returns None

Methods

__aenter__

  • async __aenter__(): Self

  • Subscribe to events and start collecting statistics.

    Returns Self

__aexit__

  • async __aexit__(exc_type, exc_value, exc_traceback): None

  • Stop collecting statistics.

    Parameters

    • exc_type: type[BaseException] | None
    • exc_value: BaseException | None
    • exc_traceback: TracebackType | None

    Returns None

calculate

  • calculate(): FinalStatistics

  • Calculate the current statistics.

    Returns FinalStatistics

record_request_processing_failure

  • record_request_processing_failure(request_id_or_key): None

  • Mark a request as failed.

    Parameters

    • request_id_or_key: str

    Returns None

record_request_processing_finish

  • record_request_processing_finish(request_id_or_key): None

  • Mark a request as finished.

    Parameters

    • request_id_or_key: str

    Returns None

record_request_processing_start

  • record_request_processing_start(request_id_or_key): None

  • Mark a request as started.

    Parameters

    • request_id_or_key: str

    Returns None

register_status_code

  • register_status_code(code): None

  • Increment the number of times a status code has been received.

    Parameters

    • code: int

    Returns None

reset

  • async reset(): None

  • Reset the statistics to their defaults and remove any persistent state.

    Returns None