Skip to main content
Version: Next

EnqueueUrlsOptions

Options accepted by the enqueueUrls() context helper exposed by BasicCrawler.

Hierarchy

Index

Properties

Properties

optionalbaseUrl

baseUrl?: string

A base URL that will be used to resolve relative URLs.

optionalexclude

exclude?: readonly UrlPatternInput[]

An array of URL patterns. Matching URLs will not be enqueued.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

optionalinheritedforefront

forefront?: boolean = false

If set to true:

  • while adding the request to the queue: the request will be added to the foremost position in the queue.
  • while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned in the next call to RequestQueue.fetchNextRequest. By default, it's put to the end of the queue.

In case the request is already present in the queue, this option has no effect.

If more requests are added with this option at once, their order in the following fetchNextRequest call is arbitrary.

optionalinclude

include?: readonly UrlPatternInput[]

An array of URL patterns that URLs must match to be enqueued.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

The patterns are combined with the strategy using AND logic - a URL must match at least one include pattern and satisfy the strategy to be enqueued. To match URLs across hostnames, pass an explicit EnqueueStrategy.All strategy.

If undefined, the links are enqueued based on the strategy alone. Passing an empty array is not allowed.

optionallabel

label?: string

Sets Request.label for newly enqueued requests.

Can be overwritten by transformRequestFunction.

optionallimit

limit?: number

Limit the amount of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.

optionalonSkippedRequest

onSkippedRequest?: SkippedRequestCallback

When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped

  1. based on robots.txt file,
  2. because they don't match enqueueLinks filters,
  3. or because the maxRequestsPerCrawl limit has been reached

optionalsessionId

sessionId?: string

Sets Request.sessionId for newly enqueued requests.

optionalskipNavigation

skipNavigation?: boolean = false

If set to true, tells the crawler to skip navigation and process the request directly.

optionalstrategy

strategy?: EnqueueStrategyOption = EnqueueStrategyOption

The strategy to use when enqueueing the urls.

Depending on the strategy you select, we will only check certain parts of the URLs found. Here is a diagram of each URL part and their name:

Protocol          Domain
┌────┐          ┌─────────┐
https://example.crawlee.dev/...
│       └─────────────────┤
│             Hostname    │
│                         │
└─────────────────────────┘
         Origin

optionaltransformRequestFunction

transformRequestFunction?: RequestTransform

After request options are filtered by include/exclude patterns, this function can be used to remove them or modify their contents such as userData, payload or, most importantly uniqueKey. This is useful when you need to enqueue multiple Requests to the queue that share the same URL, but differ in methods or payloads, or to dynamically update or create userData.

For example: by adding keepUrlFragment: true to the request options, URL fragments will not be removed when uniqueKey is computed.

Example:

{
    transformRequestFunction: (request) => {
        request.userData.foo = 'bar';
        request.keepUrlFragment = true;
        return request;
    }
}

Note that transformRequestFunction has the highest priority and can overwrite the global label option.

The function receives a RequestOptions object and can return either:

  • The modified RequestOptions object
  • 'unchanged' to keep the original options as-is
  • A falsy value or 'skip' to exclude the request from the queue

optionaluserData

userData?: Dictionary

Sets Request.userData for newly enqueued requests.

optionalwaitForAllRequestsToBeAdded

waitForAllRequestsToBeAdded?: boolean

By default, only the first batch (1000) of found requests will be added to the queue before resolving the call. You can use this option to wait for adding all of them.

Page Options