ThrottlingRequestManagerOptions <T>
Index
Properties
optionalbaseDelaySecs
The delay applied after a domain's first HTTP 429, doubled on each subsequent one.
domains
Which domains to throttle: a list of hostnames, or 'all' for every domain the crawl encounters.
Matching a listed hostname is case-insensitive and exact - wildcards such as *.example.com are not
supported, so list each subdomain you care about (or set
throttleBy: 'registrableDomain'). An
internationalized domain may be given in either its unicode or its punycode form, and an IPv6 address has
to be bracketed ([::1]). Requests for any other domain bypass throttling entirely.
'all' gives each domain a queue of its own the first time it is seen, so that it can be held back
without its requests being repeatedly popped and re-enqueued. One request queue per domain is not free,
which is what maxThrottledDomains is
there to bound.
inner
The request manager to wrap, usually a RequestQueue. Requests for domains that are not throttled are stored here.
optionalmaxDelaySecs
Upper bound on the delay between requests to a rate-limited domain, applied to both the exponential
backoff and a Retry-After value.
optionalmaxDomainStallSecs
How long a domain may rate-limit us without a single request getting through before the crawl is abandoned with a PersistentRateLimitError.
A domain that keeps answering 429 for this long is not going to be crawled by waiting longer - the
concurrency is too high for it, or it has blocked us outright. Its requests are deliberately left in
their queue, so re-running the crawl with purgeOnStart disabled picks them up once the domain recovers.
A crawler running with keepAlive is exempt - outliving a domain that will not let us through is the
whole point there.
optionalmaxThrottledDomains
The most domains a run may throttle at once. Exceeding it throws, rather than silently letting the throttling lapse - one request queue per domain is not free, and a crawl that discovers domains without bound would drown the storage backend in them.
Only domains discovered under domains: 'all' count against this; an explicit list is taken at face value.
optionalminCrawlDelaySecs
A floor under the crawl delay of every throttled domain, in seconds - the proactive clock described on
ThrottlingRequestManager. A domain whose robots.txt asks for a longer Crawl-delay gets the
longer one; this is a minimum, not an override.
optionalpersistStateKey
The key under which the discovered domain list is kept in the default key-value store, so that a restart
with purgeOnStart disabled reopens their queues instead of stranding whatever they still hold. Only
written under domains: 'all'.
Give each manager its own key when running several of them against the same storage.
optionalrequestManagerOpener
Opens the per-domain queues, one per throttled domain, each under the alias throttled-<domain>.
optionalthrottleBy
What counts as "the same domain": the exact hostname, or the registrable domain it belongs to
(example.com for www.example.com, a.example.co.uk and so on). Hosts with no registrable domain -
IP addresses, localhost - are always throttled per hostname.
Grouping by registrable domain gives subdomains a single pair of clocks and a single queue, which is what you want when the pacing is there to be polite to one server rather than to satisfy a specific host's rate limit.
Options for ThrottlingRequestManager.