SupportsDomainThrottling

A request manager that can pace requests per domain, as ThrottlingRequestManager does.

The crawlers detect this structurally rather than by type, so a wrapper can opt in by forwarding these three methods without IRequestManager having to know that throttling exists.

Implemented by

• ThrottlingRequestManager

Index

Methods

• assertNoStalledDomains
• recordDomainDelay
• setCrawlDelay

Methods

assertNoStalledDomains

• assertNoStalledDomains(): Promise<void>

• ---

  Returns Promise<void>

  See more at ThrottlingRequestManager.assertNoStalledDomains

recordDomainDelay

• recordDomainDelay(url, retryAfterMs): boolean

• ---

  Parameters

  • url: string

  • optionalretryAfterMs: null | number

  Returns boolean

  See more at ThrottlingRequestManager.recordDomainDelay

setCrawlDelay

• setCrawlDelay(url, delaySeconds): boolean

• ---

  Parameters

  • url: string

  • delaySeconds: number

  Returns boolean

  See more at ThrottlingRequestManager.setCrawlDelay