IRequestLoader
Hierarchy
- IRequestLoader
Implemented by
Index
Methods
[asyncIterator]
Can be used to iterate over the loader instance in a
for await .. ofloop. Provides an alternative for the repeated use offetchNextRequest.Returns AsyncGenerator<CrawleeRequest<Dictionary>, any, any>
fetchNextRequest
Gets the next Request to process, or
nullif there are no more pending requests.The returned request is marked as in progress and remains so until it is passed to IRequestLoader.markRequestAsHandled. The caller is responsible for eventually marking every fetched request as handled; otherwise the loader never considers itself finished and the request may be re-served after a restart. See the request lifecycle contract on IRequestLoader.
Returns Promise<null | CrawleeRequest<T>>
getHandledCount
Returns the number of requests in the loader that have been handled.
Returns Promise<number>
getPendingCount
Returns an approximation of the number of pending requests in the loader.
Returns Promise<number>
getTotalCount
Returns an approximation of the total number of requests in the loader (i.e. pending + handled).
Returns Promise<number>
isEmpty
Resolves to
trueif the next call to IRequestLoader.fetchNextRequest function would returnnull, otherwise it resolves tofalse. Note that even if the loader is empty, there might be some pending requests currently being processed.This is a statement about what the next fetch would return, not about how much work is left, so it may report
truewhile IRequestLoader.getPendingCount is non-zero - a loader that withholds requests for a while (as ThrottlingRequestManager does for a rate-limited domain) is empty for as long as it will not hand anything over. UseisFinished()to ask whether the work is done.Returns Promise<boolean>
isFinished
Returns
trueif all requests were already handled and there are no more left.Returns Promise<boolean>
markRequestAsHandled
Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.
Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.isFinished from ever resolving to
trueand skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.Parameters
request: CrawleeRequest<Dictionary>
Returns Promise<null | void | RequestQueueOperationInfo>
optionalpersistState
Persists the current state of the loader into the default KeyValueStore.
Not all loaders support persistence; implementations that do not should leave this
undefined.Returns Promise<void>
optionaltoTandem
Combines the loader with a request manager to support adding and reclaiming requests.
Parameters
optionalrequestManager: IRequestManager
Request manager to combine the loader with. If not provided, the default RequestQueue is used.
Returns Promise<IRequestManager>
An abstract interface defining a read-only stream of requests to crawl.
Request loaders are used to manage and provide access to a storage of crawling requests.
Key responsibilities:
Request lifecycle contract
Every request returned by IRequestLoader.fetchNextRequest is considered in progress until it is passed to IRequestLoader.markRequestAsHandled. Once you fetch a request, you are obligated to eventually mark it as handled — there is no way to hand a request back to a loader (only an IRequestManager can reclaim requests for a retry). "Handled" therefore means "finished with this request", whether processing succeeded or was abandoned after exhausting retries.
Honoring this contract matters for three reasons:
trueonce nothing is in progress. Leaving a request unmarked keeps the crawler running indefinitely.Concrete implementations such as RequestList or SitemapRequestLoader build on this interface. The IRequestManager interface extends it with the capability to enqueue and reclaim requests.