Skip to main content
Version: 4.0 (RC)

Result Storage

Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the CRAWLEE_STORAGE_DIR environment variable. If this variable is not defined, by default Crawlee sets CRAWLEE_STORAGE_DIR to ./storage in the current working directory.

By default, Crawlee storage is managed by the FileSystemStorageBackend class, which stores all information as local files in the respective storage type folders.

Key-value store​

The key-value store is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.

Each Crawlee project run is associated with a default key-value store. By convention, the project input and output are stored in the default key-value store under the INPUT and OUTPUT keys respectively. Typically, both input and output are JSON files, although they could be any other format.

In Crawlee, the key-value store is represented by the KeyValueStore class. In order to simplify access to the default key-value store, Crawlee also provides KeyValueStore.getValue() and KeyValueStore.setValue() functions.

The data is stored in the directory specified by the CRAWLEE_STORAGE_DIR environment variable as follows:

{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
note

{STORE_ID} is the name or the ID of the key-value store. The default key-value store has ID default, unless we override it by setting the CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID environment variable. The {KEY} is the key of the record and {EXT} corresponds to the MIME content type of the data value.

The following code demonstrates basic operations of key-value stores:

import { KeyValueStore } from 'crawlee';

// Get the INPUT from the default key-value store
const input = await KeyValueStore.getInput();

// Write the OUTPUT to the default key-value store
await KeyValueStore.setValue('OUTPUT', { myResult: 123 });

// Open a named key-value store
const store = await KeyValueStore.open('some-name');

// Write a record to the named key-value store.
// JavaScript object is automatically converted to JSON,
// strings and binary buffers are stored as they are
await store.setValue('some-key', { foo: 'bar' });

// Read a record from the named key-value store.
// Note that JSON is automatically parsed to a JavaScript object,
// text data is returned as a string, and other data is returned as binary buffer
const value = await store.getValue('some-key');

// Delete a record from the named key-value store
await store.setValue('some-key', null);

To see a real-world example of how to get the input from the key-value store, see the Screenshots example.

Dataset​

Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. Dataset can be imagined as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - we can only add new records to it, but we cannot modify or remove existing records.

Each Crawlee project run is associated with a default dataset. Typically, it is used to store crawling results specific for the crawler run. Its usage is optional.

In Crawlee, the dataset is represented by the Dataset class. In order to simplify writes to the default dataset, Crawlee also provides the Dataset.pushData() function.

The data is stored in the directory specified by the CRAWLEE_STORAGE_DIR environment variable as follows:

{CRAWLEE_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json
note

{DATASET_ID} is the name or the ID of the dataset. The default dataset has ID default, unless we override it by setting the CRAWLEE_DEFAULT_DATASET_ID environment variable. Each dataset item is stored as a separate JSON file, where {INDEX} is a zero-based index of the item in the dataset.

The following code demonstrates basic operations of the dataset:

import { Dataset } from 'crawlee';

// Write a single row to the default dataset
await Dataset.pushData({ col1: 123, col2: 'val2' });

// Open a named dataset
const dataset = await Dataset.open('some-name');

// Write a single row
await dataset.pushData({ foo: 'bar' });

// Write multiple rows
await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);

To see how to use the dataset to store crawler results, see the Cheerio Crawler example.

Cleaning up the storages​

Run-scoped storages - the default one and any opened with an alias - are purged before the crawler starts if not specified otherwise. Storages opened with a name persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via Dataset.open()) or when we try to work with a default storage via one of the helper methods (e.g. Dataset.pushData() that under the hood calls Dataset.open()). If we don't work with storages explicitly in our code, the purging will eventually happen when the run method of our crawler is executed. In case we need to purge the storages sooner, we can use the purgeDefaultStorages() helper explicitly:

import { purgeDefaultStorages } from 'crawlee';

await purgeDefaultStorages();

Calling this function will clean up the run-scoped results storage directories - the default ones and any alias-keyed one - except the INPUT key in the default key-value store directory. This is a shortcut for running (optional) purge method on the StorageBackend interface, in other words it will call the purge method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.

Transactional storage​

A request handler either finishes, or it doesn't. Without further care, a handler that pushes a few dataset items, updates a key-value record and then throws would leave those writes behind — and when the request is retried, they would happen again, duplicating your data.

Crawlee prevents that by wrapping every request in a storage transaction. Writes made through the storage classes (Dataset, KeyValueStore, RequestQueue) and the context helpers (pushData, enqueueLinks, ...) while a request is being handled are recorded rather than applied. When the request handler succeeds, they are replayed into real storage all together; when it throws, they are dropped. A retry therefore never double-writes, and a failed request leaves no partial results behind.

This is on by default for every crawler and covers the whole request lifecycle — preNavigationHooks, postNavigationHooks, extendContext and the request handler alike.

What you can rely on​

  • Atomicity per request. A request's storage writes become visible all together on success, or not at all on failure.
  • Read-your-own-writes. A read that follows a write in the same handler sees the written value — getData(), getValue(), key listings, dataset iteration and getInfo() all merge the handler's buffered writes with the real storage contents.
  • Isolation of uncommitted writes. One handler's uncommitted writes are invisible to concurrently running handlers — to pass data between handlers, use useState() instead of the key-value store.
  • Fidelity. Values are captured at write time (via structuredClone), so mutating an object after passing it to pushData() or setValue() affects neither what the handler reads back nor what gets committed.

What the transaction deliberately does not cover​

  • Request queue additions (by default). Adding requests is idempotent (the queue deduplicates by uniqueKey), and buffering them would starve the crawl of new work until the handler finishes. New requests are therefore applied immediately (the writeThrough policy) and are not rolled back when the handler fails. If you need strict all-or-nothing enqueues, opt into buffering:

    const crawler = new CheerioCrawler({
        transactionalStorage: { requestQueue: 'deferred' },
        // ...
    });

    Under the deferred policy, the requestId returned by an addRequests() call inside a handler is provisional — the real id is assigned by the storage backend when the transaction commits.

  • Shared state. The object returned by useState() / KeyValueStore.getAutoSavedValue() is the sanctioned live channel shared by all handlers, and it stays live: mutations of it are not rolled back when a handler fails.

  • Cross-storage atomicity. The commit spans multiple storages and multiple calls; delivery is at-least-once. If a commit fails partway, the request fails and is retried, and the retry may re-apply what already landed.

  • Error handlers. errorHandler and failedRequestHandler run after the failed request's transaction has been rolled back, so their writes go straight to real storage — that is what error handlers are for.

  • Deferred cleanups. Callbacks registered with registerDeferredCleanup() run after the transaction is closed, so their writes land immediately and are not rolled back. In AdaptivePlaywrightCrawler they run once per request handler attempt, so a write there can land twice for one request — push your results from the request handler instead.

Escape hatches​

The feature is escapable at three granularities:

  • Whole feature: transactionalStorage: false on the crawler options disables the mechanism entirely; every storage call behaves exactly as if no transactions existed. (Not supported by AdaptivePlaywrightCrawler, which needs per-attempt buffering to work at all.)

  • Per storage type: the transactionalStorage: { requestQueue: 'writeThrough' | 'deferred' } policy object, described above.

  • Per call site: withDirectStorageAccess() runs a callback outside the transaction, so its writes land immediately and are never rolled back. Use it for progress files, streaming output, and anything that genuinely must not wait for the handler to succeed:

    import { withDirectStorageAccess } from 'crawlee';

    async function requestHandler({ request }) {
        await withDirectStorageAccess(async () => {
            const store = await KeyValueStore.open();
            await store.setValue(`progress-${request.id}`, { startedAt: new Date() });
        });
        // ... the rest of the handler is transactional as usual
    }

Operations that throw inside a transaction​

A few operations cannot be buffered, and silently letting them through would produce storage states that no rollback can undo. They throw inside a transaction, with an error pointing at withDirectStorageAccess():

  • Dataset.drop(), KeyValueStore.drop(), RequestQueue.drop() and RequestQueue.purge(),
  • the request queue processing internals (fetchNextRequest(), markRequestAsHandled(), reclaimRequest()),
  • KeyValueStore.setValue() with a stream value — a stream can only be consumed once, so it cannot serve both a read within the handler and the commit replay. Write streams under withDirectStorageAccess().

Programmatic use​

Outside a crawler, storage is non-transactional unless you ask for it explicitly:

import { withStorageTransaction, Dataset } from 'crawlee';

await withStorageTransaction(async () => {
    const dataset = await Dataset.open();
    await dataset.pushData({ some: 'data' }); // buffered
    // committed when the callback returns, rolled back if it throws
});

For full control (e.g. deciding whether to commit only after inspecting the results), use createStorageTransaction() and drive run() / commit() / rollback() / dispose() yourself. This is exactly what AdaptivePlaywrightCrawler does with its per-attempt transactions.