Skip to main content
Version: 1.10

_DomainState

Tracks delay state for a single domain.

Index

Methods

Properties

Methods

apply_backoff

  • apply_backoff(now, delay): None

  • Block the domain for delay.

    If no 429 arrives for another delay after the domain becomes dispatchable again, the exponent resets.

    Parameters

    • now: datetime
    • delay: timedelta

    Returns None

apply_crawl_delay

  • apply_crawl_delay(now): None

  • Block the domain for its crawl-delay, if it declared one.

    Parameters

    • now: datetime

    Returns None

is_stall_candidate

  • is_stall_candidate(now, window): bool

  • Check whether the domain still rate-limits us and has rate-limited every request for longer than window.

    Parameters

    • now: datetime
    • window: timedelta

    Returns bool

reset_throttling

  • reset_throttling(): None

  • Clear the transient throttle state.

    Returns None

Properties

backoff_decays_at

backoff_decays_at: datetime

Time after which an incoming 429 is treated as a fresh burst rather than a continuation of the current one.

backoff_until

backoff_until: datetime

Earliest time the next request is allowed by the 429 backoff. Kept apart from crawl_delay_until, which is armed on every dispatch and would otherwise pass for an active backoff.

consecutive_429_count

consecutive_429_count: int

Number of consecutive 429 responses (for exponential backoff).

crawl_delay

crawl_delay: timedelta | None

Minimum interval between requests, used to push crawl_delay_until on dispatch.

crawl_delay_until

crawl_delay_until: datetime

Earliest time the next request is allowed by the domain's crawl-delay.

domain

domain: str

The domain being tracked.

last_rate_limited_at

last_rate_limited_at: datetime | None

Time of the most recent 429 from the domain.

rate_limited_since

rate_limited_since: datetime | None

Time of the first 429 since the domain last completed a request.

throttled_until

throttled_until: datetime

Earliest time the next request to this domain is allowed by either of its two independent clocks.

Page Options