Statistics

A class for collecting, tracking, and logging runtime statistics for requests.

It is designed to record information such as request durations, retries, successes, and failures, enabling analysis of crawler performance. The collected statistics are persisted to a KeyValueStore, ensuring they remain available across crawler migrations, abortions, and restarts. This persistence allows for tracking and evaluation of crawler behavior over its lifecycle.

Hierarchy

• Statistics
  • _NonPersistentStatistics

Index

Methods

• __aenter__
• __aexit__
• __init__
• calculate
• record_request_processing_failure
• record_request_processing_finish
• record_request_processing_start
• register_status_code
• replace_state_model
• reset
• with_default_state

Properties

• active

Methods

__aenter__

• async __aenter__(): Self

• Subscribe to events and start collecting statistics.

  ---

  Returns Self

__aexit__

• async __aexit__(exc_type, exc_value, exc_traceback): None

• Stop collecting statistics.

  ---

  Parameters

  • exc_type: type[BaseException] | None

  • exc_value: BaseException | None

  • exc_traceback: TracebackType | None

  Returns None

__init__

• __init__(*, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval, state_model): None

• Parameters

  • optionalkeyword-onlypersistence_enabled: bool = False

  • optionalkeyword-onlypersist_state_kvs_name: str = 'default'

  • optionalkeyword-onlypersist_state_key: str | None = None

  • optionalkeyword-onlykey_value_store: KeyValueStore | None = None

  • optionalkeyword-onlylog_message: str = 'Statistics'

  • optionalkeyword-onlyperiodic_message_logger: Logger | None = None

  • optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)

  • keyword-onlystate_model: type[TStatisticsState]

  Returns None

calculate

• calculate(): FinalStatistics

• Calculate the current statistics.

  ---

  Returns FinalStatistics

record_request_processing_failure

• record_request_processing_failure(request_id_or_key): None

• Mark a request as failed.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

record_request_processing_finish

• record_request_processing_finish(request_id_or_key): None

• Mark a request as finished.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

record_request_processing_start

• record_request_processing_start(request_id_or_key): None

• Mark a request as started.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

register_status_code

• register_status_code(code): None

• Increment the number of times a status code has been received.

  ---

  Parameters

  • code: int

  Returns None

replace_state_model

• replace_state_model(state_model): Statistics[TNewStatisticsState]

• Create near copy of the Statistics with replaced state_model.

  ---

  Parameters

  • state_model: type[TNewStatisticsState]

  Returns Statistics[TNewStatisticsState]

reset

• async reset(): None

• Reset the statistics to their defaults and remove any persistent state.

  ---

  Returns None

with_default_state

• with_default_state(*, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval): Statistics[StatisticsState]

• Convenience constructor for creating a Statistics with default state model StatisticsState.

  ---

  Parameters

  • optionalkeyword-onlypersistence_enabled: bool = False

  • optionalkeyword-onlypersist_state_kvs_name: str = 'default'

  • optionalkeyword-onlypersist_state_key: str | None = None

  • optionalkeyword-onlykey_value_store: KeyValueStore | None = None

  • optionalkeyword-onlylog_message: str = 'Statistics'

  • optionalkeyword-onlyperiodic_message_logger: Logger | None = None

  • optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)

  Returns Statistics[StatisticsState]

Properties

active

active: bool

Indicate whether the context is active.