Statistics

A class for collecting, tracking, and logging runtime statistics for requests.

It is designed to record information such as request durations, retries, successes, and failures, enabling analysis of crawler performance. The collected statistics are persisted to a KeyValueStore, ensuring they remain available across crawler migrations, abortions, and restarts. This persistence allows for tracking and evaluation of crawler behavior over its lifecycle.

Hierarchy

• Statistics
  • _NonPersistentStatistics

Index

Methods

• __aenter__
• __aexit__
• __init__
• calculate
• record_request_processing_failure
• record_request_processing_finish
• record_request_processing_start
• register_status_code
• replace_state_model
• reset
• with_default_state

Properties

• active

Methods

__aenter__

• async __aenter__(): Self

• Overrides Statistics.__aenter__

  Subscribe to events and start collecting statistics.

  ---

  Returns Self

__aexit__

• async __aexit__(exc_type, exc_value, exc_traceback): None

• Overrides Statistics.__aexit__

  Stop collecting statistics.

  ---

  Parameters

  • exc_type: type[BaseException] | None

  • exc_value: BaseException | None

  • exc_traceback: TracebackType | None

  Returns None

__init__

• __init__(*, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval, state_model, statistics_log_format): None

• Overrides Statistics.__init__

  Parameters

  • optionalkeyword-onlypersistence_enabled: bool = False

  • optionalkeyword-onlypersist_state_kvs_name: str | None = None

  • optionalkeyword-onlypersist_state_key: str | None = None

  • optionalkeyword-onlykey_value_store: KeyValueStore | None = None

  • optionalkeyword-onlylog_message: str = 'Statistics'

  • optionalkeyword-onlyperiodic_message_logger: Logger | None = None

  • optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)

  • keyword-onlystate_model: type[TStatisticsState]

  • optionalkeyword-onlystatistics_log_format: Literal[table, inline] = 'table'

  Returns None

calculate

• calculate(): FinalStatistics

• Inherited from Statistics.calculate

  Calculate the current statistics.

  ---

  Returns FinalStatistics

record_request_processing_failure

• record_request_processing_failure(request_id_or_key): None

• Inherited from Statistics.record_request_processing_failure

  Mark a request as failed.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

record_request_processing_finish

• record_request_processing_finish(request_id_or_key): None

• Inherited from Statistics.record_request_processing_finish

  Mark a request as finished.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

record_request_processing_start

• record_request_processing_start(request_id_or_key): None

• Inherited from Statistics.record_request_processing_start

  Mark a request as started.

  ---

  Parameters

  • request_id_or_key: str

  Returns None

register_status_code

• register_status_code(code): None

• Inherited from Statistics.register_status_code

  Increment the number of times a status code has been received.

  ---

  Parameters

  • code: int

  Returns None

replace_state_model

• replace_state_model(state_model): Statistics[TNewStatisticsState]

• Inherited from Statistics.replace_state_model

  Create near copy of the Statistics with replaced state_model.

  ---

  Parameters

  • state_model: type[TNewStatisticsState]

  Returns Statistics[TNewStatisticsState]

reset

• async reset(): None

• Inherited from Statistics.reset

  Reset the statistics to their defaults and remove any persistent state.

  ---

  Returns None

with_default_state

• with_default_state(*, persistence_enabled, persist_state_kvs_name, persist_state_key, key_value_store, log_message, periodic_message_logger, log_interval, statistics_log_format): Statistics[StatisticsState]

• Inherited from Statistics.with_default_state

  Initialize a new instance with default state model StatisticsState.

  ---

  Parameters

  • optionalkeyword-onlypersistence_enabled: bool = False

  • optionalkeyword-onlypersist_state_kvs_name: str | None = None

  • optionalkeyword-onlypersist_state_key: str | None = None

  • optionalkeyword-onlykey_value_store: KeyValueStore | None = None

  • optionalkeyword-onlylog_message: str = 'Statistics'

  • optionalkeyword-onlyperiodic_message_logger: Logger | None = None

  • optionalkeyword-onlylog_interval: timedelta = timedelta(minutes=1)

  • optionalkeyword-onlystatistics_log_format: Literal[table, inline] = 'table'

  Returns Statistics[StatisticsState]

Properties

active

active: bool

Indicate whether the context is active.