Skip to main content
Version: Next

CrawlerAddRequestsOptions

Options accepted by the enqueueUrls() context helper exposed by BasicCrawler.

Hierarchy

Index

Properties

Properties

optionalinheritedbaseUrl

baseUrl?: string

A base URL that will be used to resolve relative URLs.

optionalinheritedbatchSize

batchSize?: number = 1000

optionalinheritedexclude

exclude?: readonly UrlPatternInput[]

An array of URL patterns. Matching URLs will not be enqueued.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

optionalinheritedforefront

forefront?: boolean = false

If set to true:

  • while adding the request to the queue: the request will be added to the foremost position in the queue.
  • while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned in the next call to RequestQueue.fetchNextRequest. By default, it's put to the end of the queue.

In case the request is already present in the queue, this option has no effect.

If more requests are added with this option at once, their order in the following fetchNextRequest call is arbitrary.

optionalinheritedinclude

include?: readonly UrlPatternInput[]

An array of URL patterns that URLs must match to be enqueued.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

The patterns are combined with the strategy using AND logic - a URL must match at least one include pattern and satisfy the strategy to be enqueued. To match URLs across hostnames, pass an explicit EnqueueStrategy.All strategy.

If undefined, the links are enqueued based on the strategy alone. Passing an empty array is not allowed.

optionalinheritedlabel

label?: string

Sets Request.label for newly enqueued requests.

Can be overwritten by transformRequestFunction.

optionalinheritedlimit

limit?: number

Limit the amount of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.

optionalinheritedmaxNewRequests

maxNewRequests?: number

If set, only this many actually new requests (i.e. not already present in the queue) will be added. Once the budget is reached, remaining requests from the iterable will be collected in requestsOverLimit instead.

This is useful in combination with maxRequestsPerCrawl to avoid duplicate URLs consuming the budget.

Note: Setting this option implicitly enables waitForAllRequestsToBeAdded, since all batches must complete before leftover requests can be accurately reported.

optionalinheritedonSkippedRequest

onSkippedRequest?: SkippedRequestCallback

When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped

  1. based on robots.txt file,
  2. because they don't match enqueueLinks filters,
  3. or because the maxRequestsPerCrawl limit has been reached

optionalinheritedsessionId

sessionId?: string

Sets Request.sessionId for newly enqueued requests.

optionalinheritedskipNavigation

skipNavigation?: boolean = false

If set to true, tells the crawler to skip navigation and process the request directly.

optionalinheritedstrategy

strategy?: EnqueueStrategyOption = EnqueueStrategyOption

The strategy to use when enqueueing the urls.

Depending on the strategy you select, we will only check certain parts of the URLs found. Here is a diagram of each URL part and their name:

Protocol          Domain
┌────┐          ┌─────────┐
https://example.crawlee.dev/...
│       └─────────────────┤
│             Hostname    │
│                         │
└─────────────────────────┘
         Origin

optionalinheritedtransformRequestFunction

transformRequestFunction?: RequestTransform

After request options are filtered by include/exclude patterns, this function can be used to remove them or modify their contents such as userData, payload or, most importantly uniqueKey. This is useful when you need to enqueue multiple Requests to the queue that share the same URL, but differ in methods or payloads, or to dynamically update or create userData.

For example: by adding keepUrlFragment: true to the request options, URL fragments will not be removed when uniqueKey is computed.

Example:

{
    transformRequestFunction: (request) => {
        request.userData.foo = 'bar';
        request.keepUrlFragment = true;
        return request;
    }
}

Note that transformRequestFunction has the highest priority and can overwrite the global label option.

The function receives a RequestOptions object and can return either:

  • The modified RequestOptions object
  • 'unchanged' to keep the original options as-is
  • A falsy value or 'skip' to exclude the request from the queue

optionalinheriteduserData

userData?: Dictionary

Sets Request.userData for newly enqueued requests.

optionalinheritedwaitBetweenBatchesMillis

waitBetweenBatchesMillis?: number = 1000

optionalinheritedwaitForAllRequestsToBeAdded

waitForAllRequestsToBeAdded?: boolean = false

Whether to wait for all the provided requests to be added, instead of waiting just for the initial batch of up to batchSize.

Page Options