Skip to main content
Version: Next

RequestQueueBackend

Operations on a single request queue.

A backend implementation owns all request bookkeeping (pending, in-progress, handled). Any coordination required between multiple distributed clients accessing the same queue (e.g. request locking on the Apify platform) is an internal concern of the implementation and is not exposed on this interface.

Index

Methods

Methods

addBatchOfRequests

  • Add a batch of requests to the queue.

    Each request is deduplicated by its uniqueKey. Duplicates are reported in the result but not re-added. With forefront, requests are placed at the beginning of the queue so they are processed sooner.

    Parameters

    Returns Promise<BatchAddRequestsResult>

drop

  • drop(): Promise<void>

  • Remove the request queue and all its data.

    Returns Promise<void>

fetchNextRequest

  • Return the next request in the queue to be processed, or undefined if there are currently no pending requests.

    The returned request is marked as in-progress; it will not be returned again until it is either reclaimed via reclaimRequest or marked as handled via markRequestAsHandled.

    An undefined return value does not mean processing is finished — only that there are no pending requests right now. Use isEmpty (together with the frontend's knowledge of pending add operations) to determine whether the queue is truly finished.

    Returns Promise<undefined | UpdateRequestSchema>

getMetadata

  • Returns metadata about the request queue (id, name, timestamps, request counts, etc.).

    Implementations should throw if the underlying storage no longer exists (e.g. it was deleted externally). This method should never return stale data for a storage that has been removed.

    Returns Promise<RequestQueueInfo>

getRequest

  • Retrieve a request from the queue by its uniqueKey, or undefined if it does not exist.

    Parameters

    • uniqueKey: string

    Returns Promise<undefined | UpdateRequestSchema>

isEmpty

  • isEmpty(): Promise<boolean>

  • Resolves to true if the next call to fetchNextRequest would return undefined — i.e. there are no pending requests to fetch right now.

    Requests that are currently in progress (fetched but not yet handled or reclaimed, including requests locked by other clients sharing the same queue) are not counted. An empty queue therefore does not mean crawling is finished — those in-progress requests may still be reclaimed, and background tasks may still add more requests. Use isFinished to detect completion.

    Returns Promise<boolean>

isFinished

  • isFinished(): Promise<boolean>

  • Resolves to true only when there is no outstanding work left in the queue at all — i.e. there are no pending requests to fetch and no requests currently in progress (fetched but not yet handled or reclaimed, including requests locked by other clients sharing the same queue).

    This is the strong counterpart of isEmpty: a queue whose only remaining requests are in progress is empty (nothing to fetch) but not finished (that work might still be reclaimed). It is the building block for determining whether crawling is done — though a frontend may still need to account for its own pending background add operations on top of this.

    Returns Promise<boolean>

markRequestAsHandled

  • Mark a request previously returned by fetchNextRequest as handled.

    Handled requests are never returned again by fetchNextRequest. Returns information about the operation, or undefined if the request was not in progress.

    An undefined result is a no-op, not an error: the request is simply not something this client is currently processing, so nothing is changed and the request is never added to the queue as a side effect. (Marking an already-handled request is idempotent and still returns operation info with wasAlreadyHandled: true rather than undefined.)

    Parameters

    Returns Promise<undefined | QueueOperationInfo>

purge

  • purge(): Promise<void>

  • Remove all requests from the queue but keep the queue itself.

    Returns Promise<void>

reclaimRequest

  • Reclaim a failed request back to the queue so it can be processed again by a later call to fetchNextRequest. With forefront, the request is returned to the beginning of the queue. Returns information about the operation, or undefined if the request was not in progress.

    The request is expected to already be present in the queue (it should have been obtained via fetchNextRequest); reclaiming releases its lock rather than inserting it. An undefined result is a no-op, not an error: the request is simply not something this client is currently processing, so nothing is changed and the request is never added to the queue as a side effect. Use addBatchOfRequests to insert a new request.

    Parameters

    Returns Promise<undefined | QueueOperationInfo>

optionalsetExpectedRequestProcessingTimeSecs

  • setExpectedRequestProcessingTimeSecs(secs): Promise<void>

  • Tells the client how long (in seconds) a consumer expects to hold a request fetched via fetchNextRequest before marking it handled or reclaiming it — typically the consumer's request-processing timeout plus some padding.

    A client that coordinates consumers via locking uses this to keep the request reserved for at least this long, so that a long-running consumer does not have its request handed out again while it is still being processed. Clients that do not lock may ignore it.

    Parameters

    • secs: number

    Returns Promise<void>

Page Options