Request throttling

When crawling websites that enforce rate limits (HTTP 429) or specify crawl-delay in their robots.txt, you need a way to throttle requests per domain without blocking unrelated domains. The ThrottlingRequestManager provides exactly this.

Overview

The ThrottlingRequestManager wraps a RequestManager (typically a RequestQueue) and manages per-domain throttling. You specify which domains to throttle at initialization, and the manager automatically:

• Routes requests for listed domains into dedicated sub-managers at insertion time.
• Enforces delays from HTTP 429 responses (exponential backoff) and robots.txt crawl-delay directives.
• Schedules fairly by fetching from the domain that has been waiting the longest.
• Releases the concurrency slot when all configured domains are throttled, instead of holding it for the whole cooldown.

Requests for domains not in the configured list pass through to the main queue without any throttling.

Basic usage

To use request throttling, create a ThrottlingRequestManager with the domains you want to throttle and pass it as the request_manager to your crawler:

Run on

import asyncio

from crawlee.crawlers import BasicCrawler, BasicCrawlingContext
from crawlee.request_loaders import ThrottlingRequestManager
from crawlee.storages import RequestQueue


async def main() -> None:
    # Open the default request queue.
    queue = await RequestQueue.open()

    # Wrap it with ThrottlingRequestManager for specific domains. The throttler uses the
    # same storage backend as the underlying queue.
    throttler = ThrottlingRequestManager(
        queue,
        domains=['api.example.com', 'slow-site.org'],
        request_manager_opener=RequestQueue.open,
    )

    # Pass the throttler as the crawler's request manager.
    crawler = BasicCrawler(request_manager=throttler)

    @crawler.router.default_handler
    async def handler(context: BasicCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')

    # Add requests. Listed domains are routed directly to their throttled sub-managers.
    # Others go to the inner manager.
    await throttler.add_requests(
        [
            'https://api.example.com/data',
            'https://api.example.com/users',
            'https://slow-site.org/page1',
            'https://fast-site.com/page1',  # Not throttled
        ]
    )

    await crawler.run()


if __name__ == '__main__':
    asyncio.run(main())

How it works

1. Insertion-time routing: When you add requests via add_request or add_requests, each request is checked against the configured domain list. Matching requests go directly into a per-domain sub-manager; all others go to the inner manager. Each request added this way lives in exactly one store, so it is deduplicated there.

2. 429 backoff: When the crawler detects an HTTP 429 response, the ThrottlingRequestManager records an exponential backoff delay for that domain (starting at 2s, doubling up to 60s). Requests already in flight when the limit was hit are treated as a single rate-limit event, so the delay doubles once per backoff window rather than once per 429. Once the domain goes a full extra window without rate-limiting, the next 429 starts the backoff over at the initial delay. If the response includes a Retry-After header with a positive delay, that value takes priority.

3. Crawl-delay: If robots.txt specifies a crawl-delay, the manager enforces a minimum interval between requests to that domain.

4. Fair scheduling: fetch_next_request sorts available sub-managers by how long each domain has been waiting, ensuring no domain is starved.

5. Cooldown handling: While a domain is in a cooldown, its queued requests don't count as dispatchable, so the crawler's autoscaled pool idles instead of keeping a worker slot blocked. The requests still count towards completion, so the crawl waits for them and finishes only once every one has been handled.

Sub-manager storage

Each configured domain gets its own sub-manager, opened through the request_manager_opener callback under the alias throttled-<domain>. All of them are opened the first time you use the manager, so a domain that never receives a request still gets an empty store.

Opening the sub-managers up front also makes requests that a previous run left behind visible again. Whether they're resumed or discarded depends on Configuration.purge_on_start:

• With the default purge_on_start=True, the leftover requests are purged when the sub-manager opens, just like the requests in an unnamed inner queue.
• With purge_on_start=False, the leftover requests are picked up and crawled.

warning

Named storages are exempt from purge_on_start, but aliased ones aren't. If you give the inner RequestQueue a name to make it persistent, the inner queue keeps its requests across a restart while the per-domain stores are still purged. To keep the requests in both, set purge_on_start=False.

tip

The ThrottlingRequestManager is an opt-in feature. If you don't pass it to your crawler, requests are processed normally without any per-domain throttling.