ThrottlingRequestManagerOptions <T>
Index
Properties
optionalbaseDelaySecs
The delay applied after a domain's first HTTP 429, doubled on each subsequent one.
domains
Which domains to throttle: a list of hostnames, or 'all' for every domain the crawl encounters.
Matching a listed hostname is case-insensitive and exact - wildcards such as *.example.com are not
supported, so list each subdomain you care about (or set
throttleBy: 'registrableDomain'). An
internationalized domain may be given in either its unicode or its punycode form, and an IPv6 address has
to be bracketed ([::1]). Requests for any other domain bypass throttling entirely.
'all' gives each domain a queue of its own the first time it is seen, so that it can be held back
without its requests being repeatedly popped and re-enqueued. One request queue per domain is not free,
which is what maxThrottledDomains is
there to bound.
optionalinner
The request manager to wrap, usually a RequestQueue. Requests for domains that are not throttled are stored here. May be a factory, so that the throttler can be constructed synchronously and the manager under it opened only on first use.
Omitted, the default request queue is opened on first use through
requestManagerOpener.
optionalmaxDelaySecs
Upper bound on the delay between requests to a rate-limited domain, applied to both the exponential
backoff and a Retry-After value.
optionalmaxDomainStallSecs
How long a domain may rate-limit us without a single request getting through before the crawl is abandoned with a PersistentRateLimitError.
A domain that keeps answering 429 for this long is not going to be crawled by waiting longer - the
concurrency is too high for it, or it has blocked us outright. Its requests are deliberately left in
their queue, so re-running the crawl with purgeOnStart disabled picks them up once the domain recovers.
A crawler running with keepAlive is exempt - outliving a domain that will not let us through is the
whole point there.
optionalmaxThrottledDomains
The most domains a run may throttle at once. Exceeding it throws, rather than silently letting the throttling lapse - one request queue per domain is not free, and a crawl that discovers domains without bound would drown the storage backend in them.
Only domains discovered under domains: 'all' count against this; an explicit list is taken at face value.
optionalminCrawlDelaySecs
A floor under the crawl delay of every throttled domain, in seconds - the proactive clock described on
ThrottlingRequestManager. A domain whose robots.txt asks for a longer Crawl-delay gets the
longer one; this is a minimum, not an override. A minIntervalEverywhere PacingSignal raises
this floor at runtime, and never lowers it.
optionalpersistStateKey
The key under which the discovered domain list is kept in the default key-value store, so that a restart
with purgeOnStart disabled reopens their queues instead of stranding whatever they still hold. Only
written under domains: 'all'.
Give each manager its own key when running several of them against the same storage.
optionalrequestManagerOpener
Opens the per-domain queues, one per throttled domain, each under the alias throttled-<domain>.
optionalthrottleBy
What counts as "the same domain": the exact hostname, or the registrable domain it belongs to
(example.com for www.example.com, a.example.co.uk and so on). Hosts with no registrable domain -
IP addresses, localhost - are always throttled per hostname.
Grouping by registrable domain gives subdomains a single pair of clocks and a single queue, which is what you want when the pacing is there to be polite to one server rather than to satisfy a specific host's rate limit.
Options for ThrottlingRequestManager.