Skip to main content
Version: 4.0 (RC)

Request Storage

Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the CRAWLEE_STORAGE_DIR environment variable. If this variable is not defined, by default Crawlee sets CRAWLEE_STORAGE_DIR to ./storage in the current working directory.

Request queue​

The request queue is a storage of URLs to crawl. The queue is used for the deep crawling of websites, where we start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.

Each Crawlee project run is associated with a default request queue. Typically, it is used to store URLs to crawl in the specific crawler run. Its usage is optional.

In Crawlee, the request queue is represented by the RequestQueue class.

By default, the request queue is managed by the FileSystemStorageBackend class and its data is stored in the local directory specified by the CRAWLEE_STORAGE_DIR environment variable as follows:

{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
note

{QUEUE_ID} is the name or ID of the request queue. The default queue has ID default, unless we override it by setting the CRAWLEE_DEFAULT_REQUEST_QUEUE_ID environment variable.

note

entries.json contains an array of requests.

The following code demonstrates the usage of the request queue:

import { CheerioCrawler } from 'crawlee';

// The crawler will automatically process requests from the queue.
// It's used the same way for Puppeteer/Playwright crawlers.
const crawler = new CheerioCrawler({
    // Note that we're not specifying the requestQueue here
    async requestHandler({ enqueueLinks }) {
        // Add new request to the queue
        await crawler.addRequests([{ url: 'https://example.com/new-page' }]);
        // Add links found on page to the queue
        await enqueueLinks();
    },
});

// Add the initial requests.
// Note that we are not opening the request queue explicitly before
await crawler.addRequests([
    { url: 'https://example.com/1' },
    { url: 'https://example.com/2' },
    { url: 'https://example.com/3' },
    // ...
]);

// Run the crawler
await crawler.run();

To see more detailed example of how to use the request queue with a crawler, see the Puppeteer Crawler example.

The request queue is not optimized for adding numerous URLs in a single batch — historically, requests were added one by one. To enqueue a large set of initial URLs efficiently, use the addRequests() method (or simply pass the URLs to crawler.run()), which adds requests in batches:

// This is the suggested way.
// Note that we are not using the request list at all,
// and not using the request queue explicitly here.

import { PuppeteerCrawler } from 'crawlee';

// Prepare the sources array with URLs to visit (it can contain millions of URLs)
const sources = [
    { url: 'http://www.example.com/page-1' },
    { url: 'http://www.example.com/page-2' },
    { url: 'http://www.example.com/page-3' },
    // ...
];

// The crawler will automatically process requests from the queue.
// It's used the same way for Cheerio/Playwright crawlers
const crawler = new PuppeteerCrawler({
    async requestHandler({ enqueueLinks }) {
        // Add new request to the queue
        await crawler.addRequests(['http://www.example.com/new-page']);

        // Add links found on page to the queue
        await enqueueLinks();

        // The requests above would be added to the queue
        // and would be processed after the initial requests are processed.
    },
});

// Add the initial sources array to the request queue
// and run the crawler
await crawler.run(sources);

Reading requests from other sources​

Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides request loaders for these read-only sources — RequestList and SitemapRequestLoader — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.

See the dedicated Request loaders guide for details on loaders, request managers, and how to combine them with a queue into a RequestManagerTandem.

Cleaning up the storages​

Run-scoped storages - the default one and any opened with an alias - are purged before the crawler starts if not specified otherwise. Storages opened with a name persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via RequestQueue.open()) or when we try to work with a default storage via one of the helper methods (e.g. crawler.addRequests() that under the hood calls RequestQueue.open()). If we don't work with storages explicitly in our code, the purging will eventually happen when the run method of our crawler is executed. In case we need to purge the storages sooner, we can use the purgeDefaultStorages() helper explicitly:

import { purgeDefaultStorages } from 'crawlee';

await purgeDefaultStorages();

Calling this function will clean up the run-scoped request storage directories - the default queue and any alias-keyed one - along with the request list stored in the default key-value store. This is a shortcut for running (optional) purge method on the StorageBackend interface, in other words it will call the purge method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set onlyPurgeOnce to true in the options object.