IRequestManager

Represents a provider of requests/URLs to crawl.

Implemented by

• RequestManagerTandem
• RequestProvider

Index

Methods

• [asyncIterator]
• addRequest
• addRequestsBatched
• fetchNextRequest
• getPendingCount
• getTotalCount
• handledCount
• isEmpty
• isFinished
• markRequestHandled
• reclaimRequest

Methods

[asyncIterator]

• [asyncIterator](): AsyncGenerator<Request<Dictionary>, any, any>

• Can be used to iterate over the RequestManager instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

  ---

  Returns AsyncGenerator<Request<Dictionary>, any, any>

addRequest

• addRequest(requestLike, options): Promise<RequestQueueOperationInfo>

• Parameters

  • requestLike: Source

  • optionaloptions: RequestQueueOperationOptions

  Returns Promise<RequestQueueOperationInfo>

addRequestsBatched

• addRequestsBatched(requests, options): Promise<AddRequestsBatchedResult>

• Parameters

  • requests: RequestsLike

  • optionaloptions: AddRequestsBatchedOptions

  Returns Promise<AddRequestsBatchedResult>

fetchNextRequest

• fetchNextRequest<T>(): Promise<null | Request<T>>

• Gets the next Request to process.

  The function's Promise resolves to null if there are no more requests to process.

  ---

  Returns Promise<null | Request<T>>

getPendingCount

• getPendingCount(): number

• Get an offline approximation of the number of pending requests.

  ---

  Returns number

getTotalCount

• getTotalCount(): number

• Get the total number of requests known to the request manager.

  ---

  Returns number

handledCount

• handledCount(): Promise<number>

• Returns number of handled requests.

  ---

  Returns Promise<number>

isEmpty

• isEmpty(): Promise<boolean>

• Resolves to true if the next call to IRequestManager.fetchNextRequest function would return null, otherwise it resolves to false. Note that even if the provider is empty, there might be some pending requests currently being processed.

  ---

  Returns Promise<boolean>

isFinished

• isFinished(): Promise<boolean>

• Returns true if all requests were already handled and there are no more left.

  ---

  Returns Promise<boolean>

markRequestHandled

• markRequestHandled(request): Promise<null | void | RequestQueueOperationInfo>

• Marks request as handled after successful processing.

  ---

  Parameters

  • request: Request<Dictionary>

  Returns Promise<null | void | RequestQueueOperationInfo>

reclaimRequest

• reclaimRequest(request, options): Promise<null | RequestQueueOperationInfo>

• Reclaims request to the provider if its processing failed. The request will become available in the next fetchNextRequest().

  ---

  Parameters

  • request: Request<Dictionary>

  • optionaloptions: RequestQueueOperationOptions

  Returns Promise<null | RequestQueueOperationInfo>