Skip to main content
Version: 3.11

Statistics

The statistics class provides an interface to collecting and logging run statistics for requests.

All statistic information is saved on key value store under the key SDK_CRAWLER_STATISTICS_*, persists between migrations and abort/resurrect

Index

Properties

Methods

Properties

errorTracker

errorTracker: ErrorTracker

An error tracker for final retry errors.

errorTrackerRetry

errorTrackerRetry: ErrorTracker

An error tracker for retry errors prior to the final retry.

readonlyid

id: number = ...

Statistic instance id.

readonlyrequestRetryHistogram

requestRetryHistogram: number[] = []

Contains the current retries histogram. Index 0 means 0 retries, index 2, 2 retries, and so on

state

state: StatisticState

Current statistic state used for doing calculations on Statistics.calculate calls

Methods

calculate

  • calculate(): { crawlerRuntimeMillis: number; requestAvgFailedDurationMillis: number; requestAvgFinishedDurationMillis: number; requestTotalDurationMillis: number; requestsFailedPerMinute: number; requestsFinishedPerMinute: number; requestsTotal: number }

  • Calculate the current statistics

    Returns { crawlerRuntimeMillis: number; requestAvgFailedDurationMillis: number; requestAvgFinishedDurationMillis: number; requestTotalDurationMillis: number; requestsFailedPerMinute: number; requestsFinishedPerMinute: number; requestsTotal: number }

    • crawlerRuntimeMillis: number
    • requestAvgFailedDurationMillis: number
    • requestAvgFinishedDurationMillis: number
    • requestTotalDurationMillis: number
    • requestsFailedPerMinute: number
    • requestsFinishedPerMinute: number
    • requestsTotal: number

persistState

  • persistState(options): Promise<void>

  • Persist internal state to the key value store

    Parameters

    • optionaloptions: PersistenceOptions

      Override the persistence options provided in the constructor

    Returns Promise<void>

registerStatusCode

  • registerStatusCode(code): void

  • Increments the status code counter.

    Parameters

    • code: number

    Returns void

reset

  • reset(): void

  • Set the current statistic instance to pristine values

    Returns void

resetStore

  • resetStore(options): Promise<void>

  • Parameters

    • optionaloptions: PersistenceOptions

      Override the persistence options provided in the constructor

    Returns Promise<void>

startCapturing

  • startCapturing(): Promise<void>

  • Initializes the key value store for persisting the statistics, displaying the current state in predefined intervals

    Returns Promise<void>

stopCapturing

  • stopCapturing(): Promise<void>

  • Stops logging and remove event listeners, then persist

    Returns Promise<void>

toJSON

  • Make this class serializable when called with JSON.stringify(statsInstance) directly or through keyValueStore.setValue('KEY', statsInstance)

    Returns StatisticPersistedState

Page Options