Skip to main content
Version: Next

ThrottlingRequestManager <T>

A request manager that wraps another one and paces requests per domain.

Requests for a throttled domain are routed into their own queue when they are added, so each request lives in exactly one place and deduplication keeps working. Everything else goes to the wrapped manager untouched.

fetchNextRequest() serves the domain that has been waiting longest and skips any that are backing off, falling back to the wrapped manager. It never blocks: while every remaining request belongs to a throttled domain it returns null and ThrottlingRequestManager.isEmpty reports true, so the crawler idles instead of holding a concurrency slot open.

Each throttled domain runs two independent clocks, and may be dispatched to once both have run out:

  • Backoff, set by HTTP 429 responses - honouring Retry-After, and otherwise doubling from baseDelaySecs. Reactive and temporary: it decays once the domain stops turning us away. The crawlers report the 429s themselves; a request held back this way is retried later without counting against maxRequestRetries and without penalising its session.
  • Crawl delay, the minimum interval between two dispatches to the domain, armed after each one. Proactive and constant: whatever the domain's robots.txt asks for, floored by minCrawlDelaySecs. Either may be absent, in which case the other one is the delay.

Which domains get those clocks is domains - a list, or 'all' for every domain the crawl encounters.

Example usage:

const crawler = new CheerioCrawler({
    requestManager: new ThrottlingRequestManager({
        inner: await RequestQueue.open(),
        domains: ['api.example.com', 'slow-site.org'],
    }),
    requestHandler: async ({ request }) => { ... },
});

Implements

Index

Constructors

Accessors

Methods

Constructors

constructor

Accessors

innerManager

  • get innerManager(): T

  • The wrapped manager, holding every request whose domain is not throttled.

    Returns T

Methods

[asyncIterator]

  • Can be used to iterate over the loader instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<CrawleeRequest<Dictionary>, void, unknown>

addRequest

addRequestsBatched

  • Adds requests in batches, routing each one to the manager that owns its domain.

    Batching, validation, deduplication and Retry-After-free bookkeeping are all delegated to the target managers - this only decides where each request goes, one batch at a time, so a lazy or unbounded input iterable is never fully materialized.

    Parameters

    Returns Promise<AddRequestsBatchedResult>

assertNoStalledDomains

  • assertNoStalledDomains(): Promise<void>

  • Throws PersistentRateLimitError if any domain has been rate-limiting us past maxDomainStallSecs without letting a single request through.

    A domain qualifies only while it still has queued requests and is actively rate-limiting - a domain that has simply run out of work is finished, not stalled, and one being waited out under a long robots.txt Crawl-delay is being obeyed, not stonewalled.

    Returns Promise<void>

drop

  • drop(): Promise<void>

  • Returns Promise<void>

fetchNextRequest

  • Returns the next request from a domain that is not backing off, or from the inner manager.

    Returns null while every remaining request belongs to a throttled domain - it never waits the backoff out, because a consumer parked in here holds a concurrency slot, which the autoscaler reads as spare capacity and answers by scaling up. Callers poll instead, and ThrottlingRequestManager.isEmpty reports true meanwhile so the crawler's task loop idles rather than spins.

    Returns Promise<null | CrawleeRequest<R>>

getHandledCount

  • getHandledCount(): Promise<number>

  • Returns the number of requests in the loader that have been handled.

    Returns Promise<number>

getPendingCount

  • getPendingCount(): Promise<number>

  • Returns an approximation of the number of pending requests in the loader.

    Returns Promise<number>

getTotalCount

  • getTotalCount(): Promise<number>

  • Returns an approximation of the total number of requests in the loader (i.e. pending + handled).

    Returns Promise<number>

isEmpty

  • isEmpty(): Promise<boolean>

  • Whether the next ThrottlingRequestManager.fetchNextRequest would return null.

    Requests waiting on a throttled domain count as unavailable, so a crawler whose task loop is gated on this idles for the backoff instead of spinning on a fetch that cannot succeed yet.

    Returns Promise<boolean>

isFinished

  • isFinished(): Promise<boolean>

  • Unlike ThrottlingRequestManager.isEmpty, throttled requests still count as outstanding work.

    Returns Promise<boolean>

markRequestAsHandled

  • Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.

    Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.isFinished from ever resolving to true and skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.

    Parameters

    Returns Promise<null | void | RequestQueueOperationInfo>

persistState

  • persistState(): Promise<void>

  • Persists the current state of the loader into the default KeyValueStore.

    Not all loaders support persistence; implementations that do not should leave this undefined.

    Returns Promise<void>

purge

  • purge(): Promise<void>

  • Empties every manager and clears the accumulated backoff. A robots.txt Crawl-delay is a property of the site rather than of the run, so it survives.

    Returns Promise<void>

purgeDomainQueues

  • purgeDomainQueues(): Promise<void>

  • Empties the per-domain queues, leaving the wrapped manager alone.

    Those queues are this manager's own no matter who owns the one it wraps, which is what makes this safe to call where a full purge() would not be.

    Returns Promise<void>

reclaimRequest

recordDomainDelay

  • recordDomainDelay(url, retryAfterMs): boolean

  • Records a 429 response and puts the URL's domain into backoff.

    Parameters

    • url: string
    • optionalretryAfterMs: null | number

    Returns boolean

    false if the domain is not configured for throttling, in which case this is a no-op.

setCrawlDelay

  • setCrawlDelay(url, delaySeconds): boolean

  • Records the Crawl-delay a domain's robots.txt asked for, which becomes its crawl delay unless minCrawlDelaySecs asks for longer.

    The first value wins, so a robots.txt re-fetch cannot change the cadence mid-crawl.

    Parameters

    • url: string
    • delaySeconds: number

    Returns boolean

    false if the domain is not throttled, in which case this is a no-op.

setExpectedRequestProcessingTimeSecs

  • setExpectedRequestProcessingTimeSecs(secs): Promise<void>

  • Tells the manager how long a consumer expects to hold a request fetched via fetchNextRequest() before marking it handled or reclaiming it (typically the request-handler timeout plus padding).

    Managers backed by a storage backend that reserves requests via locking use this to avoid handing the same request out again while it is still being processed. Implementations that do not need this hint may leave it undefined.

    Parameters

    • secs: number

    Returns Promise<void>

Page Options