Skip to main content
Version: 4.0 (RC)

Request loaders

Request loaders extend the functionality of the RequestQueue, providing additional tools for managing URLs and requests. If you are new to Crawlee and unfamiliar with the RequestQueue, consider starting with the Request storage guide first. Request loaders define how requests are fetched and stored, enabling various use cases such as reading URLs from a static list, a sitemap, an external API, or combining multiple sources together.

Overview​

The request loader abstractions are built around two interfaces and a couple of helpers:

  • IRequestLoader: The base interface for reading requests in a crawl.
  • IRequestManager: Extends IRequestLoader with write capabilities (adding and reclaiming requests), and with the pacing signals a crawler reports back.
  • RequestManagerTandem: Combines a read-only IRequestLoader with a writable IRequestManager.
  • ThrottlingRequestManager: Wraps a writable IRequestManager and paces requests per domain.

And the concrete request loader implementations:

Below is a class diagram that illustrates the relationships between these components and the RequestQueue:

Crawler usage

A crawler reads its requests from a single IRequestManager, passed via the requestManager option. A RequestQueue is itself a request manager, so it can be passed directly. A read-only loader (such as RequestList) cannot — combine it with a queue into a tandem first, see the Request manager tandem section below.

Request loaders​

The IRequestLoader interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and reporting whether the loader has a request ready, is waiting on one, or is done. It is intentionally read-only — it does not allow adding new requests. Concrete implementations such as RequestList build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.

Request list​

The RequestList manages a static list of URLs to crawl. The list is created for a single crawler run and, unlike a queue, cannot have requests added to or removed from it after initialization. It can hold a large number of URLs (even millions) with significantly lower overhead than enqueueing them one by one.

Here is a basic example of working with the RequestList:

import { RequestList } from 'crawlee';

// Open a request list with a static set of URLs.
// The name is used to persist the list's state in the default key-value store.
const requestList = await RequestList.open('my-list', [
    'https://crawlee.dev/',
    'https://crawlee.dev/docs',
    'https://crawlee.dev/api',
]);

// Iterate over the requests manually (a crawler does this for you under the hood).
for await (const request of requestList) {
    console.log(request.url);
    await requestList.markRequestAsHandled(request);
}

Sitemap request loader​

The SitemapRequestLoader is a specialized request loader that reads URLs from sitemaps following the Sitemaps protocol. It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.

note

The SitemapRequestLoader is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader — those should be handled by regular crawlers using the enqueueLinks functionality.

The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs.

import { SitemapRequestLoader } from 'crawlee';

// Open a sitemap request list. The sitemap is fetched and parsed in the background,
// so crawling can start before the whole sitemap is loaded.
const sitemapRequestLoader = await SitemapRequestLoader.open({
    sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
    // Optionally filter the URLs read from the sitemap:
    // include: ['https://crawlee.dev/docs/**'],
});

for await (const request of sitemapRequestLoader) {
    console.log(request.url);
    await sitemapRequestLoader.markRequestAsHandled(request);
}

Request managers​

The IRequestManager interface extends IRequestLoader with write capabilities. In addition to reading requests, a request manager can add new requests and reclaim failed ones. This is essential for dynamic crawling, where new URLs emerge during the crawl, or when requests fail and need to be retried. The RequestQueue is the primary built-in request manager — see the Request storage guide for details.

Pacing signals​

A manager decides when each request goes out, so it is also where a crawler reports back what a site said about the pace it wants to be crawled at. That arrives through one method, recordPacingSignal(): a source refused a request because we were going too fast (reason: 'rateLimited', optionally carrying the wait it asked for), or a source declared a standing floor on how often it may be requested (reason: 'minInterval', carrying that interval). Nothing in the payload names HTTP status codes, response headers or robots.txt — where a signal came from is the crawler's business, not the manager's.

Every signal also carries a scope: how much of the URL space it covers. 'hostname' and 'registrableDomain' are what Crawlee's own reporters send, and the type suggests them, but it accepts any string — so a pacer keyed on something other than a host, an account or an API key say, can be reported to in its own vocabulary. A manager may apply a signal to a wider scope than it was given, since a floor that holds for one host still holds when a whole site is paced by it, but never to a narrower one, which would leave part of what the signal covers running unpaced.

A third reason, minIntervalEverywhere, is a floor under the pace of every domain the manager dispatches to, declared by whoever owns the crawl rather than by a source — which is why it is the one variant carrying no url. It is how a crawler offers a manager its sameDomainDelaySecs: whatever paces takes the floor, and only when nothing does the crawler add a pacer of its own.

Two things follow for implementors. The method is required, so reporting is never a question of support: a manager that does not pace — a plain queue — returns false and the crawler warns that the signal was dropped, while one that wraps another forwards it, as RequestManagerTandem does, or a nested pacer goes deaf. And forwarding needs no knowledge of the payload, which is why this is one method taking a value — and why what a signal applies to travels inside that value.

Per-domain throttling​

Some sites answer bursts of traffic with HTTP 429 (Too Many Requests) rather than an outright block. By default a 429 is treated as a blocked session: the session is retired and the request is retried straight away on a fresh one, which churns through proxies without actually slowing down.

The ThrottlingRequestManager handles it at the scheduling layer instead. List the domains you want paced:

import { CheerioCrawler, ThrottlingRequestManager } from 'crawlee';

const crawler = new CheerioCrawler({
    requestManager: new ThrottlingRequestManager({
        domains: ['api.example.com'],
        // optional, these are the defaults
        baseDelaySecs: 2,
        maxDelaySecs: 60,
        maxDomainStallSecs: 900,
    }),
    requestHandler: async ({ request }) => {
        // ...
    },
});

Requests for domains it does not pace go to the default request queue, opened on first use. Pass inner to wrap a manager of your own instead — a queue you opened, or a tandem over a requestList.

The pacer works wherever you put it: pass it to the crawler directly as above, or nest it inside a tandem as the writable side of a loader — the tandem forwards the crawler's pacing signals to the manager it wraps, so listed domains are paced either way.

Requests for a listed domain are routed into their own queue as they are added. When one of those domains answers with a 429, the crawler honours its Retry-After header — or backs off exponentially from baseDelaySecs up to maxDelaySecs if there is none — and holds that domain's requests back for the duration. Requests for every other domain keep flowing at full speed, the throttled request is retried later without counting against maxRequestRetries, and its session is left alone, because a rate limit says nothing about the session.

Because a throttled request costs no retries, a domain that never stops rate-limiting would otherwise keep the crawl alive forever. If one goes maxDomainStallSecs without letting a single request through, the crawl shuts down with a PersistentRateLimitError — at that point the concurrency is too high for that domain, or it has blocked you outright, and waiting longer will not help. Its requests are left in their queue on purpose, so re-running the crawl with purgeOnStart disabled resumes them if the rate limit lifts. A crawler running with keepAlive is exempt, since staying up regardless is what it was asked to do.

Matching is exact and case-insensitive, with no wildcard support, so list each subdomain you care about — or set throttleBy: 'registrableDomain', which groups a site and all of its subdomains under a single set of clocks. That grouping is also the finest granularity this manager can pace at, since holding a domain back means holding its queue back: a pacing signal scoped more narrowly is applied to the whole group, and one scoped more widely throws rather than being quietly under-applied.

The two clocks​

Every throttled domain runs two of them, and is dispatched to once both have run out:

  • Backoff — reactive and temporary, set by HTTP 429 responses and decaying once the domain stops turning you away.
  • Crawl delay — proactive and constant, a minimum interval between two dispatches, armed after each one. It is whatever the domain's robots.txt asks for (with respectRobotsTxtFile enabled), floored by minCrawlDelaySecs. A domain asking for longer gets longer.

Throttling every domain​

domains: 'all' gives a clock to each domain the crawl encounters instead of a fixed list, and a queue of its own the first time it is seen — so a domain that is waiting out a delay is skipped rather than repeatedly fetched and put back:

const crawler = new CheerioCrawler({
    requestManager: new ThrottlingRequestManager({
        domains: 'all',
        minCrawlDelaySecs: 1,
        throttleBy: 'registrableDomain',
    }),
    requestHandler: async ({ request }) => {
        // ...
    },
});

This is what the crawler's own sameDomainDelaySecs option is built on — with no manager of your own to report the floor to, it wraps the crawler's request manager in a ThrottlingRequestManager configured exactly like the above. Pass a manager like this one yourself and it takes the floor instead, so your configuration keeps the crawl to one clock per domain. Dropping minCrawlDelaySecs is just as useful: 429 backoff and robots.txt Crawl-delay then apply to every domain, with no pacing of your own on top.

A queue per domain is not free, so a run may only throttle maxThrottledDomains of them (100 by default) before it throws. If you are crawling more domains than that, pace the crawl with maxRequestsPerMinute instead. The list of domains discovered so far is kept in the default key-value store, under persistStateKey, so that a restart reopens their queues rather than leaving whatever they still hold uncrawled.

Request manager tandem​

The RequestManagerTandem class combines the read-only capabilities of an IRequestLoader (like RequestList) with the read-write capabilities of an IRequestManager (like RequestQueue). This is useful when you need to load initial requests from a static source (such as a file, sitemap, or database) and also dynamically add or retry requests during the crawl.

Under the hood, the tandem checks whether the read-only loader still has pending requests. If so, each request from the loader is transferred to the manager (the queue) before being processed. Any newly added or reclaimed requests go directly to the manager side. Because every request passes through the queue, deduplication and retries are handled consistently and a single URL is not crawled multiple times.

The easiest way to build a tandem is the toTandem() helper available on the loaders. Called without arguments, it pairs the loader with the default RequestQueue; you can also pass a specific request manager to use instead.

Request list with request queue​

This setup is useful when you have a static list of URLs to crawl, but also need to handle dynamic requests discovered during the crawl. Requests from the RequestList are processed first by being enqueued into the RequestQueue, which handles persistence and retries.

import { CheerioCrawler, RequestList } from 'crawlee';

// A static list of URLs to start from.
const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);

// `toTandem()` is a shortcut that pairs the loader with a request queue.
// Without arguments it opens the default `RequestQueue`.
const requestManager = await requestList.toTandem();

const crawler = new CheerioCrawler({
    requestManager,
    async requestHandler({ enqueueLinks }) {
        await enqueueLinks();
    },
});

await crawler.run();

Sitemap request loader with request queue​

Similarly, you can combine a SitemapRequestLoader with a RequestQueue. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.

import { CheerioCrawler, SitemapRequestLoader } from 'crawlee';

// Read the initial URLs from a sitemap.
const sitemapRequestLoader = await SitemapRequestLoader.open({
    sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
});

// Pair the loader with the default `RequestQueue` via the `toTandem()` shortcut.
const requestManager = await sitemapRequestLoader.toTandem();

const crawler = new CheerioCrawler({
    requestManager,
    async requestHandler({ enqueueLinks }) {
        await enqueueLinks();
    },
});

await crawler.run();

Conclusion​

This guide introduced the request loader abstractions: the read-only IRequestLoader, the writable IRequestManager, and the RequestManagerTandem that combines them, along with the RequestList and SitemapRequestLoader implementations. You also saw how to pair a loader with a queue using the toTandem() helper to handle both static and dynamically discovered requests, and how ThrottlingRequestManager paces requests to individual domains.

If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!