abstractRequestProvider
Hierarchy
- RequestProvider
Implements
Index
Constructors
Properties
Methods
Constructors
constructor
Parameters
options: InternalRequestProviderOptions
config: Configuration = ...
Returns RequestProvider
Properties
assumedHandledCount
assumedTotalCount
client
clientKey
readonlyconfig
id
internalTimeoutMillis
log
optionalname
requestLockSecs
timeoutSecs
Methods
addRequest
Parameters
requestLike: Source
Request object or vanilla object with request data. Note that the function sets the
uniqueKeyandidfields to the passed Request.optionaloptions: RequestQueueOperationOptions = {}
Request queue operation options.
Returns Promise<RequestQueueOperationInfo>
addRequests
Adds requests to the queue in batches of 25. This method will wait till all the requests are added to the queue before resolving. You should prefer using
queue.addRequestsBatched()orcrawler.addRequests()if you don't want to block the processing, as those methods will only wait for the initial 1000 requests, start processing right after that happens, and continue adding more in the background.If a request passed in is already present due to its
uniqueKeyproperty being the same, it will not be updated. You can find out whether this happened by finding the request in the resulting BatchAddRequestsResult object.Parameters
requestsLike: Source[]
Request objects or vanilla objects with request data. Note that the function sets the
uniqueKeyandidfields to the passed requests if missing.optionaloptions: RequestQueueOperationOptions = {}
Request queue operation options.
Returns Promise<BatchAddRequestsResult>
addRequestsBatched
Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in the background. You can configure the batch size via
batchSizeoption and the sleep time in between the batches viawaitBetweenBatchesMillis. If you want to wait for all batches to be added to the queue, you can use thewaitForAllRequestsToBeAddedpromise you get in the response object.Parameters
requests: (string | Source)[]
The requests to add
options: AddRequestsBatchedOptions = {}
Options for the request queue
Returns Promise<AddRequestsBatchedResult>
drop
Removes the queue either from the Apify Cloud storage or from the local database, depending on the mode of operation.
Returns Promise<void>
abstractfetchNextRequest
Returns a next request in the queue to be processed, or
nullif there are no more pending requests.Once you successfully finish processing of the request, you need to call RequestQueue.markRequestHandled to mark the request as handled in the queue. If there was some error in processing the request, call RequestQueue.reclaimRequest instead, so that the queue will give the request to some other consumer in another call to the
fetchNextRequestfunction.Note that the
nullreturn value doesn't mean the queue processing finished, it means there are currently no pending requests. To check whether all requests in queue were finished, use RequestQueue.isFinished instead.Parameters
optionaloptions: RequestOptions<Dictionary>
Returns Promise<null | Request<T>>
Returns the request object or
nullif there are no more pending requests.
getInfo
Returns an object containing general information about the request queue.
The function returns the same object as the Apify API Client's getQueue function, which in turn calls the Get request queue API endpoint.
Example:
{
id: "WkzbQMuFYuamGv3YF",
name: "my-queue",
userId: "wRsJZtadYvn4mBZmm",
createdAt: new Date("2015-12-12T07:34:14.202Z"),
modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
accessedAt: new Date("2015-12-14T08:36:13.202Z"),
totalRequestCount: 25,
handledRequestCount: 5,
pendingRequestCount: 20,
}Returns Promise<undefined | RequestQueueInfo>
getRequest
Gets the request from the queue specified by ID.
Parameters
id: string
ID of the request.
Returns Promise<null | Request<T>>
Returns the request object, or
nullif it was not found.
getTotalCount
Returns an offline approximation of the total number of requests in the queue (i.e. pending + handled).
Survives restarts and actor migrations.
Returns number
handledCount
Returns the number of handled requests.
This function is just a convenient shortcut for:
const { handledRequestCount } = await queue.getInfo();Returns Promise<number>
isEmpty
Resolves to
trueif the next call to RequestQueue.fetchNextRequest would returnnull, otherwise it resolves tofalse. Note that even if the queue is empty, there might be some pending requests currently being processed. If you need to ensure that there is no activity in the queue, use RequestQueue.isFinished.Returns Promise<boolean>
isFinished
Resolves to
trueif all requests were already handled and there are no more left. Due to the nature of distributed storage used by the queue, the function might occasionally return a false negative, but it will never return a false positive.Returns Promise<boolean>
markRequestHandled
Marks a request that was previously returned by the RequestQueue.fetchNextRequest function as handled after successful processing. Handled requests will never again be returned by the
fetchNextRequestfunction.Parameters
request: Request<Dictionary>
Returns Promise<null | RequestQueueOperationInfo>
reclaimRequest
Reclaims a failed request back to the queue, so that it can be returned for processing later again by another call to RequestQueue.fetchNextRequest. The request record in the queue is updated using the provided
requestparameter. For example, this lets you store the number of retries or error messages for the request.Parameters
request: Request<Dictionary>
options: RequestQueueOperationOptions = {}
Returns Promise<null | RequestQueueOperationInfo>
staticopen
Opens a request queue and returns a promise resolving to an instance of the RequestQueue class.
RequestQueue represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud. The queue is used for deep crawling of websites, where you start with several URLs and then recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
For more details and code examples, see the RequestQueue class.
Parameters
optionalqueueIdOrName: null | string
ID or name of the request queue to be opened. If
nullorundefined, the function returns the default request queue associated with the crawler run.optionaloptions: StorageManagerOptions = {}
Open Request Queue options.
Returns Promise<RequestProvider>
Adds a request to the queue.
If a request with the same
uniqueKeyproperty is already present in the queue, it will not be updated. You can find out whether this happened from the resulting QueueOperationInfo object.To add multiple requests to the queue by extracting links from a webpage, see the enqueueLinks helper function.