Encrypt key-value store records

Any transform on data going to and from storage — encryption, compression, redaction, caching — belongs in a wrapper around the storage backend. Pass the wrapper wherever a backend goes: the crawler's storageBackend option, the storageBackend option of Dataset.open() / KeyValueStore.open() / RequestQueue.open(), or serviceLocator.setStorageBackend().

This example encrypts every key-value store record, useState() included, and passes datasets and request queues through untouched. To transform those as well, wrap the sub-backend returned by createDatasetBackend() (pushData() / getData()) or createRequestQueueBackend() (addBatchOfRequests() and the methods that return requests) the same way.

Don't subclass KeyValueStore for this. Inside a request handler, its writes are recorded in the storage transaction's journal and replayed through setValue() again on commit, so an override would run twice. The wrapper sits below the journal: a committed write passes through it once, a rolled-back one never reaches it, and reads inside the handler see their own untransformed writes.

import { createCipheriv, createDecipheriv, randomBytes } from 'node:crypto';
import { CheerioCrawler, FileSystemStorageBackend } from 'crawlee';
import type { KeyValueStoreBackend, StorageIdentifier } from '@crawlee/types';

const secret = randomBytes(32); // load it from somewhere safe instead

function encrypt(plain: Buffer): Buffer {
    const iv = randomBytes(12);
    const cipher = createCipheriv('aes-256-gcm', secret, iv);
    const body = Buffer.concat([cipher.update(plain), cipher.final()]);
    return Buffer.concat([iv, cipher.getAuthTag(), body]);
}

function decrypt(sealed: Uint8Array): Buffer {
    const decipher = createDecipheriv('aes-256-gcm', secret, sealed.subarray(0, 12));
    decipher.setAuthTag(sealed.subarray(12, 28));
    return Buffer.concat([decipher.update(sealed.subarray(28)), decipher.final()]);
}

class EncryptingStorageBackend extends FileSystemStorageBackend {
    override async createKeyValueStoreBackend(options?: StorageIdentifier): Promise<KeyValueStoreBackend> {
        const store = await super.createKeyValueStoreBackend(options);
        const { getValue, setValue } = Object.getPrototypeOf(store) as KeyValueStoreBackend;

        // Patching the instance keeps the example short; a real implementation would wrap the backend in a class of its own.
        store.getValue = async (key) => {
            const record = await getValue.call(store, key);
            return record && { ...record, value: decrypt(new Uint8Array(record.value)) };
        };
        store.setValue = async (record) => {
            // `Response` drains every input shape a backend can receive, streams included.
            const plain = Buffer.from(await new Response(record.value as BodyInit).arrayBuffer());
            await setValue.call(store, { ...record, value: encrypt(plain) });
        };
        return store;
    }
}

const crawler = new CheerioCrawler({
    storageBackend: new EncryptingStorageBackend({ localDataDirectory: './storage' }),
    async requestHandler({ request, $, useState, getKeyValueStore }) {
        const state = await useState({ visited: 0 });
        state.visited++;
        const store = await getKeyValueStore();
        await store.setValue(request.id, { url: request.url, title: $('title').text() });
    },
});

await crawler.run(['https://crawlee.dev']);