Skip to main content
Version: Next

SitemapRequestLoader

A list of URLs to crawl parsed from a sitemap.

The loading of the sitemap is performed in the background so that crawling can start before the sitemap is fully loaded.

Implements

Index

Methods

Methods

[asyncIterator]

  • Can be used to iterate over the loader instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<CrawleeRequest<Dictionary>, void, unknown>

checkReadiness

  • Reports whether the loader has a request to hand over, is waiting on one, or is done — see RequestSourceStatus.

    A consumer's task loop is gated on this, so implementations MUST answer ready before evaluating anything else. finished may arrive late behind distributed storage, but it is never wrong.

    Returns Promise<RequestLoaderStatus>

fetchNextRequest

  • Gets the next Request to process, or null if there are no more pending requests.

    The returned request is marked as in progress and remains so until it is passed to IRequestLoader.markRequestAsHandled. The caller is responsible for eventually marking every fetched request as handled; otherwise the loader never considers itself finished and the request may be re-served after a restart. See the request lifecycle contract on IRequestLoader.

    Returns Promise<null | CrawleeRequest<Dictionary>>

getHandledCount

  • getHandledCount(): Promise<number>

  • Returns the number of requests in the loader that have been handled.

    Returns Promise<number>

getPendingCount

  • getPendingCount(): Promise<number>

  • Returns an approximation of the number of pending requests in the loader.

    Returns Promise<number>

getTotalCount

  • getTotalCount(): Promise<number>

  • Returns an approximation of the total number of requests in the loader (i.e. pending + handled).

    Returns Promise<number>

isSitemapFullyLoaded

  • isSitemapFullyLoaded(): boolean

  • Indicates whether the background processing of sitemap contents has successfully finished.

    If this is false, the background processing is either still in progress or was aborted.

    Returns boolean

markRequestAsHandled

  • markRequestAsHandled(request): Promise<void>

  • Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.

    Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.checkReadiness from ever reporting finished and skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.

    Parameters

    Returns Promise<void>

persistState

  • persistState(): Promise<void>

  • Persists the current state of the loader into the default KeyValueStore.

    Not all loaders support persistence; implementations that do not should leave this undefined.

    Returns Promise<void>

teardown

  • teardown(): Promise<void>

  • Aborts the internal sitemap loading, stops the processing of the sitemap contents and drops all the pending URLs.

    Calling fetchNextRequest() after this method will always return null.

    Returns Promise<void>

toTandem

staticopen

  • Open a sitemap and start processing it.

    Resolves to a new instance of SitemapRequestLoader, which might not be fully loaded yet - i.e. the sitemap might still be loading in the background.

    Track the loading progress using the isSitemapFullyLoaded property.

    Parameters

    Returns Promise<SitemapRequestLoader>

Page Options