Custom statistics fields

Every crawler collects run statistics — requests finished, failures, retries, durations — into a Statistics instance exposed as crawler.statistics. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.

Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in statistics.state survives a run that a plain variable would not.

Declaring custom fields

Pass stateExtension when constructing a Statistics instance, and inject it into the crawler via the statistics option. The fields become part of state, typed exactly as you declared them:

import { CheerioCrawler, Statistics } from 'crawlee';

// Declare the extra fields and their initial values. Their types flow into `statistics.state`.
const statistics = new Statistics({
    stateExtension: { defaultState: { productsFound: 0 } },
});

const crawler = new CheerioCrawler({
    statistics,
    async requestHandler({ $, enqueueLinks }) {
        statistics.state.productsFound += $('.product').length;

        await enqueueLinks();
    },
});

await crawler.run(['https://crawlee.dev']);

// The custom fields are typed on `crawler.statistics` too.
console.log(`Found ${crawler.statistics.state.productsFound} products`);

The field types flow through to crawler.statistics.state, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.

defaultState is also what reset() restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.

Validating the fields on the way back

Declaring defaultState alone means a restored record is taken at its word: whatever productsFound holds in the key-value store becomes the state, and an increment on a string that says it is a number poisons every later one. Pass a deserialize conversion to check the record before trusting it — either a plain function or, as here, a Standard Schema:

import { CheerioCrawler, Statistics } from 'crawlee';
import { z } from 'zod';

const statistics = new Statistics({
    stateExtension: {
        // A default per field means the schema is the only place the fields are declared.
        deserialize: z.object({
            productsFound: z.number().default(0),
            lastSeenAt: z.coerce.date().default(() => new Date()),
        }),
        // `Date` is not JSON - this is how it gets into the record for `z.coerce.date()` to read back.
        serialize: ({ productsFound, lastSeenAt }) => ({
            productsFound,
            lastSeenAt: lastSeenAt.toISOString(),
        }),
    },
});

const crawler = new CheerioCrawler({
    statistics,
    async requestHandler({ $ }) {
        statistics.state.productsFound += $('.product').length;
        statistics.state.lastSeenAt = new Date();
    },
});

await crawler.run(['https://crawlee.dev']);

A schema that gives every field a default doubles as the declaration of the fields, so defaultState can be omitted — one place to add a field, and the defaults cannot drift from the conversion that has to accept them back. If the record fails to validate, the custom fields start from their defaults with a warning; the built-in statistics are unaffected.

The fields have to be JSON-serializable, because the state round-trips through the key-value store. A Date or a Set comes back from that trip as a string or a plain object, so either keep to numbers, strings, booleans, null, and plain arrays and objects of those, or pair deserialize with a serialize that converts them, as lastSeenAt does above.

Injected instances are not reset

A crawler resets the statistics it built itself at the start of every run(), but never one you passed in — a supplied instance keeps whatever state it was handed. If you reuse a crawler for several runs and want the counters to start from zero each time, call statistics.reset() yourself.

Crawlers that track fields of their own

AdaptivePlaywrightCrawler tracks a few extra fields itself (how many requests were handled with and without a browser, and how many rendering type mispredictions occurred). It builds a suitable Statistics instance by default, so you only need to do anything if you want to add fields of your own — in which case your declaration has to include the crawler's fields as well:

import { AdaptivePlaywrightCrawler, adaptivePlaywrightCrawlerStatisticState, Statistics } from 'crawlee';
import { z } from 'zod';

const statistics = new Statistics({
    stateExtension: {
        // The adaptive crawler tracks fields of its own, so an injected instance has to carry them too.
        deserialize: adaptivePlaywrightCrawlerStatisticState.deserialize.extend({
            productsFound: z.number().default(0),
        }),
    },
});

const crawler = new AdaptivePlaywrightCrawler({
    statistics,
    async requestHandler({ querySelectorAll }) {
        statistics.state.productsFound += (await querySelectorAll('.product')).length;
    },
});

await crawler.run(['https://crawlee.dev']);

console.log(`Handled ${crawler.statistics.state.httpOnlyRequestHandlerRuns} requests without a browser`);

Custom statistics implementations

If you need something more than extra fields — reporting to an external metrics backend, for example — the statistics option accepts any object implementing IStatistics, not just the built-in class. The crawler drives it purely through that interface.