Skip to main content
Version: Next

RequestQueue

Represents a queue of URLs to crawl, which is used for deep crawling of websites where you start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.

Each URL is represented using an instance of the Request class. The queue can only contain unique URLs. More precisely, it can only contain Request instances with distinct uniqueKey properties. By default, uniqueKey is generated from the URL, but it can also be overridden. To add a single URL multiple times to the queue, corresponding Request objects will need to have different uniqueKey properties.

Do not instantiate this class directly, use the RequestQueue.open function instead.

RequestQueue is used by BasicCrawler, CheerioCrawler, PuppeteerCrawler and PlaywrightCrawler as a source of URLs to crawl. Unlike RequestList, RequestQueue supports dynamic adding and removing of requests. On the other hand, the queue is not optimized for operations that add or remove a large number of URLs in a batch.

Example usage:

// Open the default request queue associated with the crawler run
const queue = await RequestQueue.open();

// Open a named request queue
const queueWithName = await RequestQueue.open('some-name');

// Enqueue few requests
await queue.addRequest({ url: 'http://example.com/aaa' });
await queue.addRequest({ url: 'http://example.com/bbb' });
await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });

Implements

Index

Properties

Accessors

Methods

Properties

readonlybackend

backend: RequestQueueBackend

readonlyid

id: string

readonlylog

log: CrawleeLogger

optionalreadonlyname

name?: string

Accessors

stats

  • Backend-independent usage counters tracked for this request queue (write operations and queue-head reads issued to the underlying storage backend). Counted per backend call.

    Returns RequestQueueStats

Methods

[asyncIterator]

  • Can be used to iterate over the loader instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<CrawleeRequest<Dictionary>, void, unknown>

addRequest

  • Adds a request to the queue.

    If a request with the same uniqueKey property is already present in the queue, it will not be updated. You can find out whether this happened from the resulting QueueOperationInfo object.

    To add multiple requests to the queue by extracting links from a webpage, see the enqueueLinks helper function.

    Parameters

    • requestLike: Source

      Request object or vanilla object with request data. Note that the function sets the uniqueKey and id fields to the passed Request.

    • optionaloptions: RequestQueueOperationOptions = {}

      Request queue operation options.

    Returns Promise<RequestQueueOperationInfo>

addRequests

  • Adds requests to the queue in batches of 25. This method will wait till all the requests are added to the queue before resolving. You should prefer using queue.addRequestsBatched() or crawler.addRequests() if you don't want to block the processing, as those methods will only wait for the initial 1000 requests, start processing right after that happens, and continue adding more in the background.

    If a request passed in is already present due to its uniqueKey property being the same, it will not be updated. You can find out whether this happened by finding the request in the resulting BatchAddRequestsResult object.

    Parameters

    • requestsLike: RequestsLike

      Request objects or vanilla objects with request data. Note that the function sets the uniqueKey and id fields to the passed requests if missing.

    • optionaloptions: RequestQueueOperationOptions = {}

      Request queue operation options.

    Returns Promise<BatchAddRequestsResult>

addRequestsBatched

  • Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in the background. You can configure the batch size via batchSize option and the sleep time in between the batches via waitBetweenBatchesMillis. If you want to wait for all batches to be added to the queue, you can use the waitForAllRequestsToBeAdded promise you get in the response object.

    Parameters

    Returns Promise<AddRequestsBatchedResult>

drop

  • drop(): Promise<void>

  • Removes the queue either from the Apify Cloud storage or from the local database, depending on the mode of operation.

    Returns Promise<void>

fetchNextRequest

  • Returns a next request in the queue to be processed, or null if there are no more pending requests.

    Once you successfully finish processing of the request, you need to call RequestQueue.markRequestAsHandled to mark the request as handled in the queue. If there was some error in processing the request, call RequestQueue.reclaimRequest instead, so that the queue will give the request to some other consumer in another call to the fetchNextRequest function.

    Note that the null return value doesn't mean the queue processing finished, it means there are currently no pending requests. To check whether all requests in queue were finished, use RequestQueue.isFinished instead.

    Returns Promise<null | CrawleeRequest<T>>

    Returns the request object or null if there are no more pending requests.

getHandledCount

  • getHandledCount(): Promise<number>

  • Returns the number of requests in the loader that have been handled.

    Returns Promise<number>

getInfo

  • Returns an object containing general information about the request queue.

    Example:

    {
      id: "WkzbQMuFYuamGv3YF",
      name: "my-queue",
      createdAt: new Date("2015-12-12T07:34:14.202Z"),
      modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
      accessedAt: new Date("2015-12-14T08:36:13.202Z"),
      totalRequestCount: 25,
      handledRequestCount: 5,
      pendingRequestCount: 20,
    }

    Returns Promise<RequestQueueInfo>

    Throws - If the underlying storage no longer exists (e.g. it was deleted externally).

getPendingCount

  • getPendingCount(): Promise<number>

  • Returns the total number of pending requests in the queue.

    Survives restarts and Actor migrations.

    Returns Promise<number>

getRequest

  • Gets the request from the queue specified by its uniqueKey.

    Parameters

    • uniqueKey: string

      Unique key of the request.

    Returns Promise<null | CrawleeRequest<T>>

    Returns the request object, or null if it was not found.

getTotalCount

  • getTotalCount(): Promise<number>

  • Returns the total number of requests in the queue (i.e. pending + handled).

    Survives restarts and actor migrations.

    Returns Promise<number>

isEmpty

  • isEmpty(): Promise<boolean>

  • Resolves to true if the next call to RequestQueue.fetchNextRequest would return null, i.e. there are no pending requests to fetch right now. Otherwise it resolves to false.

    Note that even if the queue is empty, there might be some requests currently being processed (fetched but not yet handled or reclaimed). An empty queue therefore does not mean crawling is finished — those in-progress requests may still be reclaimed, and background tasks may still be adding more requests. To check whether all activity in the queue has finished, use RequestQueue.isFinished.

    Returns Promise<boolean>

isFinished

  • isFinished(): Promise<boolean>

  • Resolves to true if all requests were already handled and there are no more left — including no requests currently in progress (fetched but not yet handled or reclaimed, including requests locked by other clients sharing the same queue) and no background add operations still in flight.

    Due to the nature of distributed storage used by the queue, the function may occasionally return a false negative, but it shall never return a false positive.

    Returns Promise<boolean>

markRequestAsHandled

purge

  • purge(): Promise<void>

  • Remove all requests from the queue but keep the queue itself, resetting it so it can be reused (e.g. across multiple crawler.run() calls).

    Returns Promise<void>

reclaimRequest

setExpectedRequestProcessingTimeSecs

  • setExpectedRequestProcessingTimeSecs(secs): Promise<void>

  • Tells the queue how long a consumer expects to hold a fetched request before marking it handled or reclaiming it (typically the request-handler timeout plus padding), so that a storage backend that reserves requests via locking does not hand the same request out again while it is still being processed.

    Several consumers may share one queue (and therefore one client) in a single process, so we only ever raise the reservation duration, never lower it — otherwise a short-lived consumer could cut short the reservation of a long-lived one and have its in-flight request stolen.

    Parameters

    • secs: number

    Returns Promise<void>

staticopen

  • Opens a request queue and returns a promise resolving to an instance of the RequestQueue class.

    RequestQueue represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud. The queue is used for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.

    For more details and code examples, see the RequestQueue class.

    Parameters

    • optionalidentifier: null | string | StorageIdentifier

      ID or name of the request queue to be opened. If a string is provided, it will first be looked up as an ID; if no such storage exists, it will be treated as a name. If null or undefined, the function returns the default request queue associated with the crawler run.

    • optionaloptions: StorageOpenOptions = {}

      Open Request Queue options.

    Returns Promise<RequestQueue>

Page Options