Statistics <StateExtension, PersistedStateExtension>

The statistics class provides an interface to collecting and logging run statistics for requests.

All statistic information is saved on key value store under the key CRAWLEE_CRAWLER_STATISTICS_*, persists between migrations and abort/resurrect

Custom fields are tracked by passing stateExtension - the extra fields are then part of state, persisted and restored along with the built-in ones.

Implements

• IStatistics<StateExtension>

Index

Constructors

• constructor

Properties

• errorTracker
• errorTrackerRetry
• id

Accessors

• requestRetryHistogram
• state

Methods

• calculate
• persistState
• registerStatusCode
• reset
• resetStore
• startCapturing
• stopCapturing
• toJSON

Constructors

constructor

• new Statistics<StateExtension, PersistedStateExtension>(options): Statistics<StateExtension, PersistedStateExtension>

• Construct a statistics instance to pass to a crawler via its statistics option, e.g. to preconfigure persistence or error snapshots, share it across sequential runs, or track extra fields via state.

  ---

  Parameters

  • options: StatisticsOptions<StateExtension, PersistedStateExtension> = {}

  Returns Statistics<StateExtension, PersistedStateExtension>

Properties

readonlyerrorTracker

errorTracker: ErrorTracker

Implementation of IStatistics.errorTracker

An error tracker for final retry errors.

readonlyerrorTrackerRetry

errorTrackerRetry: ErrorTracker

Implementation of IStatistics.errorTrackerRetry

An error tracker for retry errors prior to the final retry.

readonlyid

id: string

Statistic instance id.

Accessors

requestRetryHistogram

• get requestRetryHistogram(): number[]

• Implementation of IStatistics.requestRetryHistogram

  Contains the current retries histogram. Index 0 means 0 retries, index 2, 2 retries, and so on

  ---

  Returns number[]

state

• get state(): StatisticState & StateExtension

• Implementation of IStatistics.state

  Current statistic state used for doing calculations on Statistics.calculate calls

  ---

  Returns StatisticState & StateExtension

Methods

calculate

• calculate(): CalculatedStatistics

• Implementation of IStatistics.calculate

  Calculate the current statistics

  ---

  Returns CalculatedStatistics

persistState

• persistState(): Promise<void>

• Implementation of IStatistics.persistState

  Persist internal state to the key value store.

  Statistics are bookkeeping - a store that refuses the write is worth a warning, not a failed crawl. The crawler calls this from its migration handler, where a rejection would go unhandled.

  ---

  Returns Promise<void>

registerStatusCode

• registerStatusCode(code): void

• Implementation of IStatistics.registerStatusCode

  Increments the status code counter.

  ---

  Parameters

  • code: number

  Returns void

reset

• reset(): void

• Set the current statistic instance to pristine values.

  The persisted record is left alone - use Statistics.resetStore to clear that as well.

  ---

  Returns void

resetStore

• resetStore(): Promise<void>

• Clear the persisted statistics record, leaving the in-memory state alone.

  Throws while capturing - the next PERSIST_STATE event would write the record straight back.

  ---

  Returns Promise<void>

startCapturing

• startCapturing(): Promise<void>

• Implementation of IStatistics.startCapturing

  Initializes the key value store for persisting the statistics, displaying the current state in predefined intervals

  ---

  Returns Promise<void>

stopCapturing

• stopCapturing(): Promise<void>

• Implementation of IStatistics.stopCapturing

  Stops logging and remove event listeners, then persist

  ---

  Returns Promise<void>

toJSON

• toJSON(): StatisticPersistedState & PersistedStateExtension

• Make this class serializable when called with JSON.stringify(statsInstance) directly or through keyValueStore.setValue('KEY', statsInstance)

  ---

  Returns StatisticPersistedState & PersistedStateExtension