Skip to main content
Version: Next

RequestList

Represents a static list of URLs to crawl. The URLs can be provided either in code or parsed from a text file hosted on the web. RequestList is used by BasicCrawler, CheerioCrawler, PuppeteerCrawler and PlaywrightCrawler as a source of URLs to crawl.

Each URL is represented using an instance of the Request class. The list can only contain unique URLs. More precisely, it can only contain Request instances with distinct uniqueKey properties. By default, uniqueKey is generated from the URL, but it can also be overridden. To add a single URL to the list multiple times, corresponding Request objects will need to have different uniqueKey properties. You can use the keepDuplicateUrls option to do this for you when initializing the RequestList from sources.

RequestList doesn't have a public constructor, you need to create it with the asynchronous RequestList.open function. After the request list is created, no more URLs can be added to it. Unlike RequestQueue, RequestList is static but it can contain even millions of URLs.

Note that RequestList can be used together with RequestQueue by the same crawler. In such cases, each request from RequestList is enqueued into RequestQueue first and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue). In practical terms, such a combination can be useful when there is a large number of initial URLs, but more URLs would be added dynamically by the crawler.

RequestList has an internal state where it stores information about which requests were already handled and which are in progress. The state may be automatically persisted to the default KeyValueStore by setting the persistStateKey option so that if the Node.js process is restarted, the crawling can continue where it left off. The automated persisting is launched upon receiving the persistState event that is periodically emitted by EventManager.

The internal state is closely tied to the provided sources (URLs). If the sources change on crawler restart, the state will become corrupted and RequestList will raise an exception. This typically happens when the sources is a list of URLs downloaded from the web. In such case, use the persistRequestsKey option in conjunction with persistStateKey, to make the RequestList store the initial sources to the default key-value store and load them after restart, which will prevent any issues that a live list of URLs might cause.

Basic usage:

const requestList = await RequestList.open('my-request-list', [
    'http://www.example.com/page-1',
    { url: 'http://www.example.com/page-2', method: 'POST', userData: { foo: 'bar' }},
    { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },
]);

Advanced usage:

const requestList = await RequestList.open(null, [
    // Separate requests
    { url: 'http://www.example.com/page-1', method: 'GET', headers: { ... } },
    { url: 'http://www.example.com/page-2', userData: { foo: 'bar' }},

    // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`
    // Note that all URLs must start with http:// or https://
    { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },
], {
    // Persist the state to avoid re-crawling which can lead to data duplications.
    // Keep in mind that the sources have to be immutable or this will throw an error.
    persistStateKey: 'my-state',
});

Implements

Index

Methods

Methods

[asyncIterator]

  • Can be used to iterate over the loader instance in a for await .. of loop. Provides an alternative for the repeated use of fetchNextRequest.

    Returns AsyncGenerator<CrawleeRequest<Dictionary>, void, unknown>

checkReadiness

  • Reports whether the loader has a request to hand over, is waiting on one, or is done — see RequestSourceStatus.

    A consumer's task loop is gated on this, so implementations MUST answer ready before evaluating anything else. finished may arrive late behind distributed storage, but it is never wrong.

    Returns Promise<RequestLoaderStatus>

fetchNextRequest

  • Gets the next Request to process, or null if there are no more pending requests.

    The returned request is marked as in progress and remains so until it is passed to IRequestLoader.markRequestAsHandled. The caller is responsible for eventually marking every fetched request as handled; otherwise the loader never considers itself finished and the request may be re-served after a restart. See the request lifecycle contract on IRequestLoader.

    Returns Promise<null | CrawleeRequest<Dictionary>>

getHandledCount

  • getHandledCount(): Promise<number>

  • Returns the number of requests in the loader that have been handled.

    Returns Promise<number>

getPendingCount

  • getPendingCount(): Promise<number>

  • Returns the number of pending requests in the RequestList.

    Returns Promise<number>

getState

  • Returns an object representing the internal state of the RequestList instance. Note that the object's fields can change in future releases.

    Returns RequestListState

getTotalCount

  • getTotalCount(): Promise<number>

  • Returns the total number of unique requests present in the RequestList.

    Returns Promise<number>

markRequestAsHandled

  • markRequestAsHandled(request): Promise<void>

  • Marks a request previously returned by IRequestLoader.fetchNextRequest as handled, removing it from the set of in-progress requests.

    Call this once you are done with the request — whether processing succeeded or was abandoned after exhausting retries. Because a loader cannot take a request back, marking it handled is the only way to signal completion; failing to do so prevents IRequestLoader.checkReadiness from ever reporting finished and skews the handled and pending counts. See the request lifecycle contract on IRequestLoader.

    Parameters

    Returns Promise<void>

persistState

  • persistState(): Promise<void>

  • Persists the current state of the loader into the default KeyValueStore.

    Not all loaders support persistence; implementations that do not should leave this undefined.

    Returns Promise<void>

teardown

  • teardown(): Promise<void>

  • Removes the PERSIST_STATE event listener registered during initialization and persists the current state one last time. Call this when you are done with the RequestList to avoid leaking the listener (and the requests it retains) on the shared event manager.

    Returns Promise<void>

toTandem

staticopen

  • open(listNameOrOptions, sources, options): Promise<RequestList>

  • Opens a request list and returns a promise resolving to an instance of the RequestList class that is already initialized.

    RequestList represents a list of URLs to crawl, which is always stored in memory. To enable picking up where left off after a process restart, the request list sources are persisted to the key-value store at initialization of the list. Then, while crawling, a small state object is regularly persisted to keep track of the crawling status.

    For more details and code examples, see the RequestList class.

    Example usage:

    const sources = [
        'https://www.example.com',
        'https://www.google.com',
        'https://www.bing.com'
    ];

    const requestList = await RequestList.open('my-name', sources);

    Parameters

    • listNameOrOptions: null | string | RequestListOptions

      Name of the request list to be opened, or the options object. Setting a name enables the RequestList's state to be persisted in the key-value store. This is useful in case of a restart or migration. Since RequestList is only stored in memory, a restart or migration wipes it clean. Setting a name will enable the RequestList's state to survive those situations and continue where it left off.

      The name will be used as a prefix in key-value store, producing keys such as NAME-REQUEST_LIST_STATE and NAME-REQUEST_LIST_SOURCES.

      If null, the list will not be persisted and will only be stored in memory. Process restart will then cause the list to be crawled again from the beginning. We suggest always using a name.

    • optionalsources: RequestListSource[]

      An array of sources of URLs for the RequestList. It can be either an array of strings, plain objects that define at least the url property, or an array of Request instances.

      IMPORTANT: The sources array will be consumed (left empty) after RequestList initializes. This is a measure to prevent memory leaks in situations when millions of sources are added.

      Additionally, the requestsFromUrl property may be used instead of url, which will instruct RequestList to download the source URLs from a given remote location. The URLs will be parsed from the received response. In this case you can limit the URLs using regex parameter containing regular expression pattern for URLs to be included.

      For details, see the RequestListOptions.sources

    • optionaloptions: RequestListOptions = {}

      The RequestList options. Note that the listName parameter supersedes the RequestListOptions.persistStateKey and RequestListOptions.persistRequestsKey options and the sources parameter supersedes the RequestListOptions.sources option.

    Returns Promise<RequestList>

Page Options