IStatistics <StateExtension>

The statistics surface a crawler depends on: recording per-request outcomes, tracking errors, and driving the capture lifecycle for a run. Injected via the crawler's statistics option, so a custom implementation can be plugged in without subclassing the crawler.

StateExtension describes the custom fields tracked alongside the built-in StatisticState ones - see StatisticsOptions.stateExtension.

The owned-only mutators the crawler uses to own a default it built - reset()/resetStore() - are deliberately absent: an injected instance is borrowed, and the crawler never wipes it. Those live on the concrete Statistics only.

Implemented by

• Statistics

Index

Properties

• errorTracker
• errorTrackerRetry
• requestRetryHistogram
• state

Methods

• calculate
• discardRequestRecord
• persistState
• recordRequestFailure
• recordRequestStart
• recordRequestSuccess
• registerStatusCode
• startCapturing
• stopCapturing

Properties

readonlyerrorTracker

errorTracker: ErrorTracker

Tracker for errors on the final retry of a request.

readonlyerrorTrackerRetry

errorTrackerRetry: ErrorTracker

Tracker for errors on retries prior to the final one.

readonlyrequestRetryHistogram

requestRetryHistogram: number[]

Retries histogram - index i holds the number of requests that settled after i retries.

readonlystate

state: StatisticState & StateExtension

The live statistics state the crawler reads for status messages and the final summary.

Methods

calculate

• calculate(): CalculatedStatistics

• Computes the derived aggregates (averages, per-minute rates, totals) from the current state.

  ---

  Returns CalculatedStatistics

discardRequestRecord

• discardRequestRecord(id): void

• Drops a started request without counting it as succeeded or failed (e.g. skipped by robots.txt).

  ---

  Parameters

  • id: string | number

  Returns void

optionalpersistState

• persistState(): Promise<void>

• Persists the current state to the key-value store. Optional - the crawler calls it on migration, but a backend with no persistence of its own can omit it.

  ---

  Returns Promise<void>

recordRequestFailure

• recordRequestFailure(id, retryCount): void

• Marks a started request as failed, updating the failed counters and durations.

  ---

  Parameters

  • id: string | number

  • retryCount: number

  Returns void

recordRequestStart

• recordRequestStart(id): void

• Marks a request as started, so its duration can be measured on success/failure.

  ---

  Parameters

  • id: string | number

  Returns void

recordRequestSuccess

• recordRequestSuccess(id, retryCount): void

• Marks a started request as succeeded, updating the succeeded counters and durations.

  ---

  Parameters

  • id: string | number

  • retryCount: number

  Returns void

registerStatusCode

• registerStatusCode(code): void

• Increments the counter for the given HTTP status code.

  ---

  Parameters

  • code: number

  Returns void

startCapturing

• startCapturing(): Promise<void>

• Begins a capture window: loads any persisted state, subscribes to persistence events, starts periodic logging.

  ---

  Returns Promise<void>

stopCapturing

• stopCapturing(): Promise<void>

• Ends the capture window: stops logging, unsubscribes, and persists the final state.

  ---

  Returns Promise<void>