Skip to main content

SystemStatus

crawlee._autoscaling.system_status.SystemStatus

Provides a simple interface for evaluating system resource usage from snapshots collected by Snapshotter.

This class aggregates and interprets snapshots from a Snapshotter instance to evaluate the current and historical status of system resources like CPU, memory, event loop, and client API usage. It exposes two methods get_current_system_info and get_historical_system_info. The system information is computed using a weighted average of overloaded messages in the snapshots, with the weights being the time intervals between the snapshots. Each resource is computed separately, and the system is considered as overloaded whenever at least one resource is overloaded.

get_current_system_info returns a SystemInfo data structure that represents the current status of the system. The length of the current timeframe in seconds is configurable by the max_snapshot_age option and represents the max age of snapshots to be considered for the computation.

SystemStatus.get_historical_system_info returns a SystemInfo that represents the long-term status of the system. It considers the full snapshot history available in the Snapshotter instance.

Index

Constructors

Methods

Constructors

__init__

  • __init__(snapshotter, *, max_snapshot_age, cpu_overload_threshold, memory_overload_threshold, event_loop_overload_threshold, client_overload_threshold): None

  • Creates a new instance.

    Parameters

    • snapshotter: Snapshotter
    • max_snapshot_age: timedelta = timedelta(seconds=5)keyword-only
    • cpu_overload_threshold: float = 0.4keyword-only
    • memory_overload_threshold: float = 0.2keyword-only
    • event_loop_overload_threshold: float = 0.6keyword-only
    • client_overload_threshold: float = 0.3keyword-only

    Returns None

Methods

get_current_system_info

  • get_current_system_info(): SystemInfo

  • Retrieves and evaluates the current status of system resources.

    Considers snapshots within the _max_snapshot_age timeframe and determines if the system is currently overloaded based on predefined thresholds for each resource type.

    Returns SystemInfo

get_historical_system_info

  • get_historical_system_info(): SystemInfo

  • Retrieves and evaluates the historical status of system resources.

    Considers the entire history of snapshots from the Snapshotter to assess long-term system performance and determines if the system has been historically overloaded.

    Returns SystemInfo