Skip to main content
Version: Next

IRequestManager

Extends the read-only IRequestLoader interface with the capability to enqueue new requests and reclaim failed ones.

Hierarchy

Implemented by

Index

Methods

Methods

inherited[asyncIterator]

  • Can be used to iterate over the loader instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<CrawleeRequest<Dictionary>, any, any>

addRequest

addRequestsBatched

inheritedfetchNextRequest

  • Gets the next Request to process, or null if there are no more pending requests.

    The returned request is marked as in progress and remains so until it is passed to IRequestLoader.markRequestAsHandled. The caller is responsible for eventually marking every fetched request as handled; otherwise the loader never considers itself finished and the request may be re-served after a restart. See the request lifecycle contract on IRequestLoader.

    Returns Promise<null | CrawleeRequest<T>>

inheritedgetHandledCount

  • getHandledCount(): Promise<number>

  • Returns the number of requests in the loader that have been handled.

    Returns Promise<number>

inheritedgetPendingCount

  • getPendingCount(): Promise<number>

  • Returns an approximation of the number of pending requests in the loader.

    Returns Promise<number>

inheritedgetTotalCount

  • getTotalCount(): Promise<number>

  • Returns an approximation of the total number of requests in the loader (i.e. pending + handled).

    Returns Promise<number>

inheritedisEmpty

  • isEmpty(): Promise<boolean>

  • Resolves to true if the next call to IRequestLoader.fetchNextRequest function would return null, otherwise it resolves to false. Note that even if the loader is empty, there might be some pending requests currently being processed.

    This is a statement about what the next fetch would return, not about how much work is left, so it may report true while IRequestLoader.getPendingCount is non-zero - a loader that withholds requests for a while (as ThrottlingRequestManager does for a rate-limited domain) is empty for as long as it will not hand anything over. Use isFinished() to ask whether the work is done.

    Returns Promise<boolean>

inheritedisFinished

  • isFinished(): Promise<boolean>

  • Returns true if all requests were already handled and there are no more left.

    Returns Promise<boolean>

inheritedmarkRequestAsHandled

  • Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.

    Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.isFinished from ever resolving to true and skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.

    Parameters

    Returns Promise<null | void | RequestQueueOperationInfo>

optionalinheritedpersistState

  • persistState(): Promise<void>

  • Persists the current state of the loader into the default KeyValueStore.

    Not all loaders support persistence; implementations that do not should leave this undefined.

    Returns Promise<void>

optionalpurge

  • purge(): Promise<void>

  • Remove all requests from the queue but keep the queue itself, resetting it so it can be reused (e.g. across multiple crawler.run() calls).

    Implementations that do not support purging may leave this undefined.

    Returns Promise<void>

reclaimRequest

optionalsetExpectedRequestProcessingTimeSecs

  • setExpectedRequestProcessingTimeSecs(secs): Promise<void>

  • Tells the manager how long a consumer expects to hold a request fetched via fetchNextRequest() before marking it handled or reclaiming it (typically the request-handler timeout plus padding).

    Managers backed by a storage backend that reserves requests via locking use this to avoid handing the same request out again while it is still being processed. Implementations that do not need this hint may leave it undefined.

    Parameters

    • secs: number

    Returns Promise<void>

optionalinheritedtoTandem

  • Combines the loader with a request manager to support adding and reclaiming requests.

    Parameters

    Returns Promise<IRequestManager>

Page Options