RequestManagerTandem
Implements
Index
Constructors
constructor
Parameters
requestLoader: IRequestLoader
The read-only loader to read requests from first.
requestManager: IRequestManager | () => IRequestManager | Promise<IRequestManager>
The writable manager to transfer requests into and enqueue new ones. May be passed as a factory function so that the tandem can be constructed synchronously and the manager opened lazily on first use (e.g. a lazily-opened default RequestQueue).
Returns RequestManagerTandem
Methods
[asyncIterator]
Can be used to iterate over the loader instance in a
for await .. ofloop. Provides an alternative for the repeated use offetchNextRequest.Returns AsyncGenerator<CrawleeRequest<Dictionary>, void, unknown>
addRequest
Parameters
requestLike: Source
optionaloptions: RequestQueueOperationOptions
Returns Promise<RequestQueueOperationInfo>
addRequestsBatched
Parameters
requests: RequestsLike
optionaloptions: AddRequestsBatchedOptions
Returns Promise<AddRequestsBatchedResult>
checkReadiness
Reports whether the loader has a request to hand over, is waiting on one, or is done — see RequestSourceStatus.
A consumer's task loop is gated on this, so implementations MUST answer
readybefore evaluating anything else.finishedmay arrive late behind distributed storage, but it is never wrong.Returns Promise<RequestSourceStatus>
fetchNextRequest
Gets the next Request to process, or
nullif there are no more pending requests.The returned request is marked as in progress and remains so until it is passed to IRequestLoader.markRequestAsHandled. The caller is responsible for eventually marking every fetched request as handled; otherwise the loader never considers itself finished and the request may be re-served after a restart. See the request lifecycle contract on IRequestLoader.
Returns Promise<null | CrawleeRequest<T>>
getHandledCount
Returns the number of requests in the loader that have been handled.
Returns Promise<number>
getPendingCount
Returns an approximation of the number of pending requests in the loader.
Returns Promise<number>
getTotalCount
Returns an approximation of the total number of requests in the loader (i.e. pending + handled).
Returns Promise<number>
markRequestAsHandled
Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.
Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.checkReadiness from ever reporting
finishedand skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.Parameters
request: CrawleeRequest<Dictionary>
Returns Promise<null | void | RequestQueueOperationInfo>
persistState
Persists the current state of the loader into the default KeyValueStore.
Not all loaders support persistence; implementations that do not should leave this
undefined.Returns Promise<void>
purge
Remove all requests from the queue but keep the queue itself, resetting it so it can be reused (e.g. across multiple
crawler.run()calls).Implementations that do not support purging may leave this
undefined.Returns Promise<void>
reclaimRequest
Reclaims request to the provider if its processing failed. The request will be returned by some subsequent
fetchNextRequest()call.Parameters
request: CrawleeRequest<Dictionary>
optionaloptions: RequestQueueOperationOptions
Returns Promise<null | RequestQueueOperationInfo>
recordPacingSignal
Records something said about the pace requests should go out at, so that a manager which paces its own dispatch can hold requests back.
Required rather than optional, so that a wrapping manager always forwards it and a pacer nested in a composition still receives it; a manager that does not pace returns
false.Parameters
signal: PacingSignal
Returns boolean
trueif anything in the composition took responsibility for the signal.
setExpectedRequestProcessingTimeSecs
Tells the manager how long a consumer expects to hold a request fetched via
fetchNextRequest()before marking it handled or reclaiming it (typically the request-handler timeout plus padding).Managers backed by a storage backend that reserves requests via locking use this to avoid handing the same request out again while it is still being processed. Implementations that do not need this hint may leave it
undefined.Parameters
secs: number
Returns Promise<void>
A request manager that combines a IRequestLoader (such as a
RequestList) with a writable IRequestManager (such as aRequestQueue). It first reads requests from the loader and then, when needed, transfers them in batches to the manager.