Skip to main content
Version: Next

Snapshotter

Creates snapshots of system resources at given intervals and marks the resource as either overloaded or not during the last interval. Keeps a history of the snapshots. It tracks the following resources: Memory, EventLoop, API and CPU. The class is used by the AutoscaledPool class.

When running on the Apify platform, the CPU and memory statistics are provided by the platform, as collected from the running Docker container. When running locally, Snapshotter makes its own statistics by querying the OS.

CPU becomes overloaded locally when its current use exceeds the maxUsedCpuRatio option or when Apify platform marks it as overloaded.

Memory becomes overloaded if its current use exceeds the maxUsedMemoryRatio option. It's computed using the total memory available to the container when running on the Apify platform and a quarter of total system memory when running locally. Max total memory when running locally may be overridden by using the CRAWLEE_MEMORY_MBYTES environment variable.

Event loop becomes overloaded if it slows down by more than the maxBlockedMillis option.

Client becomes overloaded when rate limit errors (429 - Too Many Requests), typically received from the request queue, exceed the set limit within the set interval.

Index

Constructors

constructor

Properties

client

clientInterval

clientInterval: BetterIntervalID = ...

clientSnapshotIntervalMillis

clientSnapshotIntervalMillis: number

clientSnapshots

clientSnapshots: ClientSnapshot[] = []

config

cpuSnapshots

cpuSnapshots: CpuSnapshot[] = []

eventLoopInterval

eventLoopInterval: BetterIntervalID = ...

eventLoopSnapshotIntervalMillis

eventLoopSnapshotIntervalMillis: number

eventLoopSnapshots

eventLoopSnapshots: EventLoopSnapshot[] = []

events

events: EventManager

lastLoggedCriticalMemoryOverloadAt

lastLoggedCriticalMemoryOverloadAt: null | Date = null

log

log: Log

maxBlockedMillis

maxBlockedMillis: number

maxClientErrors

maxClientErrors: number

maxMemoryBytes

maxMemoryBytes: number

maxUsedMemoryRatio

maxUsedMemoryRatio: number

memorySnapshots

memorySnapshots: MemorySnapshot[] = []

snapshotHistoryMillis

snapshotHistoryMillis: number

Methods

getClientSample

  • getClientSample(sampleDurationMillis?: number): ClientSnapshot[]
  • Returns a sample of latest Client snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.


    Parameters

    • optionalsampleDurationMillis: number

    Returns ClientSnapshot[]

getCpuSample

  • getCpuSample(sampleDurationMillis?: number): CpuSnapshot[]
  • Returns a sample of latest CPU snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.


    Parameters

    • optionalsampleDurationMillis: number

    Returns CpuSnapshot[]

getEventLoopSample

  • getEventLoopSample(sampleDurationMillis?: number): EventLoopSnapshot[]
  • Returns a sample of latest event loop snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.


    Parameters

    • optionalsampleDurationMillis: number

    Returns EventLoopSnapshot[]

getMemorySample

  • getMemorySample(sampleDurationMillis?: number): MemorySnapshot[]
  • Returns a sample of latest memory snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.


    Parameters

    • optionalsampleDurationMillis: number

    Returns MemorySnapshot[]

start

  • start(): Promise<void>
  • Starts capturing snapshots at configured intervals.


    Returns Promise<void>

stop

  • stop(): Promise<void>
  • Stops all resource capturing.


    Returns Promise<void>