Skip to main content
Version: 4.0 (RC)

Running multiple crawlers

A single process can run several crawlers at once. A common setup is a fast CheerioCrawler that discovers links and a PlaywrightCrawler that renders the pages it finds. Another is one crawler per target site, each with its own rate limit.

Every crawler has its own queue​

Only the first crawler created in a process uses the default request queue. Every crawler after it opens a queue of its own, so crawler.run(urls) and crawler.addRequests() never leak requests into a sibling crawler.

Sometimes we need to reference a crawler's queue ourselves, for example to check on it from outside the crawler or to keep it between runs. In that case, we open the queue with an alias and pass it as the requestManager option:

const detailQueue = await RequestQueue.open({ alias: 'details' });
const detailCrawler = new PlaywrightCrawler({ requestManager: detailQueue, requestHandler: async () => { /* ... */ } });

An alias identifies a run-scoped queue. Crawlee purges it on start along with the default storages. Opening the same alias again returns the same queue, handled requests included. Use a name instead if the queue has to survive the process. See request storage for the details.

Feeding one crawler from another​

The enqueueLinks() context helper always adds to the crawler's own queue. To feed a different crawler, extract the links with extractLinks() and pass them to that crawler's addRequests(). It takes the same include and exclude filters, and it applies the target crawler's limits, such as maxRequestsPerCrawl.

Unlike enqueueLinks(), this does not restrict links to the current hostname. extractLinks() returns every link the selector matches, and addRequests() defaults to strategy: 'all'. Pass strategy and baseUrl if you need the restriction, as the example below does.

import { CheerioCrawler, PlaywrightCrawler, RequestQueue } from 'crawlee';

// The detail crawler's finish check needs its own queue, so we open that queue ourselves.
const detailQueue = await RequestQueue.open({ alias: 'details' });

const listingCrawler = new CheerioCrawler({
    async requestHandler({ request, extractLinks, enqueueLinks }) {
        // Product pages go to the detail crawler's queue. Unlike `enqueueLinks()`, `addRequests()` keeps
        // off-site links unless we pass a strategy and the URL it is relative to.
        await detailCrawler.addRequests(await extractLinks({ selector: '.product-item > a' }), {
            strategy: 'same-hostname',
            baseUrl: request.loadedUrl ?? request.url,
        });
        // The next listing page stays in the listing crawler's own queue.
        await enqueueLinks({ selector: 'a.pagination__next' });
    },
});

const detailCrawler = new PlaywrightCrawler({
    requestManager: detailQueue,
    taskLoopOptions: {
        // An empty queue only means the detail crawler has caught up - the listing crawler may still add more.
        isFinishedFunction: async () => {
            const { status: detailQueueStatus } = await detailQueue.checkReadiness();
            return !listingCrawler.running && detailQueueStatus === 'finished';
        },
    },
    async requestHandler({ request, page, pushData }) {
        await pushData({ url: request.url, title: await page.title() });
    },
});

// Start both together: `listingCrawler.running` is `false` until its `run()` is called, so the detail crawler would otherwise finish immediately.
await Promise.all([
    listingCrawler.run(['https://warehouse-theme-metal.myshopify.com/collections/sales']),
    detailCrawler.run(),
]);

The taskLoopOptions.isFinishedFunction option makes this work. By default, a crawler finishes as soon as its queue has nothing left to process. The detail crawler starts with an empty queue, and its queue empties again every time it catches up with the listing crawler. Without the override, it would exit long before the listing crawler is done. The override waits until the listing crawler is no longer running and checkReadiness() reports 'finished'. At that point no new requests can arrive and the detail crawler has handled every request it received.

keepAlive overrides isFinishedFunction

With keepAlive: true, the crawler ignores isFinishedFunction and never finishes on its own. Don't combine the two. If the producer eventually ends, use isFinishedFunction and leave keepAlive off.

stop() is not a way to drain a queue​

crawler.stop() stops dispatching new requests, lets the in-flight ones finish and resolves run(). Anything still waiting in the queue stays there. That suits stopping once we have enough results. It does not work for ending a keepAlive crawler that another crawler feeds:

// Don't do this - whatever the detail crawler has not picked up yet is abandoned.
await listingCrawler.run(['https://warehouse-theme-metal.myshopify.com/collections/sales']);
detailCrawler.stop();

If the producer finishes at some point, drop keepAlive and use isFinishedFunction as shown above. Keep keepAlive for producers that never finish on their own, such as a crawler running in a web server.

Sharing resources between crawlers​

Crawlers in one process don't know about each other, so each one scales its concurrency and warms up its sessions independently. We can share both between crawlers: