Snapshotter

Creates snapshots of system resources at given intervals and marks the resource as either overloaded or not during the last interval. Keeps a history of the snapshots. It tracks the following resources: Memory, EventLoop, API and CPU. The class is used by the AutoscaledPool class.

When running on the Apify platform, the CPU and memory statistics are provided by the platform, as collected from the running Docker container. When running locally, Snapshotter makes its own statistics by querying the OS.

CPU becomes overloaded locally when its current use exceeds the maxUsedCpuRatio option or when Apify platform marks it as overloaded.

Memory becomes overloaded if its current use exceeds the maxUsedMemoryRatio option. It's computed using the total memory available to the container when running on the Apify platform and a quarter of total system memory when running locally. Max total memory when running locally may be overridden by using the CRAWLEE_MEMORY_MBYTES environment variable.

Event loop becomes overloaded if it slows down by more than the maxBlockedMillis option.

Client becomes overloaded when rate limit errors (429 - Too Many Requests), typically received from the request queue, exceed the set limit within the set interval.

Index

Constructors

• constructor

Properties

• client
• clientInterval
• clientSnapshotIntervalMillis
• clientSnapshots
• config
• cpuSnapshots
• eventLoopInterval
• eventLoopSnapshotIntervalMillis
• eventLoopSnapshots
• events
• lastLoggedCriticalMemoryOverloadAt
• log
• maxBlockedMillis
• maxClientErrors
• maxMemoryBytes
• maxUsedMemoryRatio
• memorySnapshots
• snapshotHistoryMillis

Methods

• getClientSample
• getCpuSample
• getEventLoopSample
• getMemorySample
• start
• stop

Constructors

constructor

• new Snapshotter(options?: SnapshotterOptions): Snapshotter

• Parameters

  • optionaloptions: SnapshotterOptions = {}

    All Snapshotter configuration options.

  Returns Snapshotter

Properties

client

client: StorageClient

clientInterval

clientInterval: BetterIntervalID = ...

clientSnapshotIntervalMillis

clientSnapshotIntervalMillis: number

clientSnapshots

clientSnapshots: ClientSnapshot[] = []

config

config: Configuration

cpuSnapshots

cpuSnapshots: CpuSnapshot[] = []

eventLoopInterval

eventLoopInterval: BetterIntervalID = ...

eventLoopSnapshotIntervalMillis

eventLoopSnapshotIntervalMillis: number

eventLoopSnapshots

eventLoopSnapshots: EventLoopSnapshot[] = []

events

events: EventManager

lastLoggedCriticalMemoryOverloadAt

lastLoggedCriticalMemoryOverloadAt: null | Date = null

log

log: Log

maxBlockedMillis

maxBlockedMillis: number

maxClientErrors

maxClientErrors: number

maxMemoryBytes

maxMemoryBytes: number

maxUsedMemoryRatio

maxUsedMemoryRatio: number

memorySnapshots

memorySnapshots: MemorySnapshot[] = []

snapshotHistoryMillis

snapshotHistoryMillis: number

Methods

getClientSample

• getClientSample(sampleDurationMillis?: number): ClientSnapshot[]

• Returns a sample of latest Client snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.

  ---

  Parameters

  • optionalsampleDurationMillis: number

  Returns ClientSnapshot[]

getCpuSample

• getCpuSample(sampleDurationMillis?: number): CpuSnapshot[]

• Returns a sample of latest CPU snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.

  ---

  Parameters

  • optionalsampleDurationMillis: number

  Returns CpuSnapshot[]

getEventLoopSample

• getEventLoopSample(sampleDurationMillis?: number): EventLoopSnapshot[]

• Returns a sample of latest event loop snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.

  ---

  Parameters

  • optionalsampleDurationMillis: number

  Returns EventLoopSnapshot[]

getMemorySample

• getMemorySample(sampleDurationMillis?: number): MemorySnapshot[]

• Returns a sample of latest memory snapshots, with the size of the sample defined by the sampleDurationMillis parameter. If omitted, it returns a full snapshot history.

  ---

  Parameters

  • optionalsampleDurationMillis: number

  Returns MemorySnapshot[]

start

• start(): Promise<void>

• Starts capturing snapshots at configured intervals.

  ---

  Returns Promise<void>

stop

• stop(): Promise<void>

• Stops all resource capturing.

  ---

  Returns Promise<void>