Stopping the crawler early

Calling crawler.stop() ends the run gracefully: no further requests are dispatched, the ones already being processed finish, and crawler.run() resolves. Requests still waiting in the queue are left there untouched, so a later run() picks up where this one left off. This is the way to stop once a result limit is reached, without draining the queue or marking the remaining requests as handled.

To abort immediately without waiting for in-flight requests, use crawler.teardown() instead.

Run on

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    async requestHandler({ request, $, enqueueLinks, pushData, useState }) {
        const state = await useState({ found: 0, limit: 10 });

        const price = $('.price').first().text();
        if (price) {
            await pushData({ url: request.url, price });
            state.found++;
        }

        // Once we have what we came for, stop dispatching new requests. Requests already in
        // flight finish normally and `crawler.run()` resolves; whatever is still queued stays queued.
        if (state.found >= state.limit) {
            crawler.stop('Collected enough prices');
            return;
        }

        await enqueueLinks();
    },
});

await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);

// Calling `run()` again with no arguments continues with the requests left in the queue.
// Here we raise the limit first, otherwise the first request would stop the crawler again.
const state = await crawler.useState({ found: 0, limit: 10 });
state.limit = 20;
await crawler.run();