Skip to main content
Version: Next

IRequestList

Represents a static list of URLs to crawl.

Implemented by

Index

Methods

Methods

[asyncIterator]

  • [asyncIterator](): AsyncGenerator<Request<Dictionary>, any, any>

  • Gets the next Request to process. First, the function gets a request previously reclaimed using the RequestList.reclaimRequest function, if there is any. Otherwise it gets the next request from sources.

    The function resolves to null if there are no more requests to process.

    Can be used to iterate over the RequestList instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<Request<Dictionary>, any, any>

fetchNextRequest

  • fetchNextRequest(): Promise<null | Request<Dictionary>>

  • Gets the next Request to process. First, the function gets a request previously reclaimed using the RequestList.reclaimRequest function, if there is any. Otherwise it gets the next request from sources.

    The function's Promise resolves to null if there are no more requests to process.

    Returns Promise<null | Request<Dictionary>>

handledCount

  • handledCount(): number

  • Returns number of handled requests.

    Returns number

isEmpty

  • isEmpty(): Promise<boolean>

  • Resolves to true if the next call to IRequestList.fetchNextRequest function would return null, otherwise it resolves to false. Note that even if the list is empty, there might be some pending requests currently being processed.

    Returns Promise<boolean>

isFinished

  • isFinished(): Promise<boolean>

  • Returns true if all requests were already handled and there are no more left.

    Returns Promise<boolean>

length

  • length(): number

  • Returns the total number of unique requests present in the list.

    Returns number

markRequestHandled

  • markRequestHandled(request: Request<Dictionary>): Promise<void>

  • Marks request as handled after successful processing.

    Parameters

    Returns Promise<void>

persistState

  • persistState(): Promise<void>

  • Persists the current state of the IRequestList into the default KeyValueStore. The state is persisted automatically in regular intervals, but calling this method manually is useful in cases where you want to have the most current state available after you pause or stop fetching its requests. For example after you pause or abort a crawl. Or just before a server migration.

    Returns Promise<void>

reclaimRequest

  • reclaimRequest(request: Request<Dictionary>): Promise<void>

  • Reclaims request to the list if its processing failed. The request will become available in the next this.fetchNextRequest().

    Parameters

    Returns Promise<void>

Page Options