EnqueueLinksOptions
Hierarchy
- RequestQueueOperationOptions
- EnqueueLinksOptions
Index
Properties
optionalbaseUrl
optionalexclude
An array of glob pattern strings, regexp patterns or plain objects containing patterns matching URLs that will never be enqueued.
The plain objects must include either the glob
property or the regexp
property.
All remaining keys will be used as request options for the corresponding enqueued Request objects.
Glob matching is always case-insensitive. If you need case-sensitive matching, provide a regexp.
optionalforefront
If set to true
:
- while adding the request to the queue: the request will be added to the foremost position in the queue.
- while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned in the next call to RequestQueue.fetchNextRequest. By default, it's put to the end of the queue.
optionalglobs
An array of glob pattern strings or plain objects containing glob pattern strings matching the URLs to be enqueued.
The plain objects must include at least the glob
property, which holds the glob pattern string.
All remaining keys will be used as request options for the corresponding enqueued Request objects.
The matching is always case-insensitive.
If you need case-sensitive matching, use regexps
property directly.
If globs
is an empty array or undefined
, and regexps
are also not defined, then the function
enqueues the links with the same subdomain.
optionallabel
Sets Request.label for newly enqueued requests.
optionallimit
Limit the amount of actually enqueued URLs to this number. Useful for testing across the entire crawling scope.
optionalpseudoUrls
NOTE: In future versions of SDK the options will be removed.
Please use globs
or regexps
instead.
An array of PseudoUrl strings or plain objects containing PseudoUrl strings matching the URLs to be enqueued.
The plain objects must include at least the purl
property, which holds the pseudo-URL string.
All remaining keys will be used as request options for the corresponding enqueued Request objects.
With a pseudo-URL string, the matching is always case-insensitive.
If you need case-sensitive matching, use regexps
property directly.
If pseudoUrls
is an empty array or undefined
, then the function
enqueues the links with the same subdomain.
optionalregexps
An array of regular expressions or plain objects containing regular expressions matching the URLs to be enqueued.
The plain objects must include at least the regexp
property, which holds the regular expression.
All remaining keys will be used as request options for the corresponding enqueued Request objects.
If regexps
is an empty array or undefined
, and globs
are also not defined, then the function
enqueues the links with the same subdomain.
optionalrequestQueue
A request queue to which the URLs will be enqueued.
optionalselector
A CSS selector matching links to be enqueued.
optionalskipNavigation
If set to true
, tells the crawler to skip navigation and process the request directly.
optionalstrategy
The strategy to use when enqueueing the urls.
Depending on the strategy you select, we will only check certain parts of the URLs found. Here is a diagram of each URL part and their name:
Protocol Domain
┌────┐ ┌─────────┐
https://example.crawlee.dev/...
│ └─────────────────┤
│ Hostname │
│ │
└─────────────────────────┘
Origin
optionaltransformRequestFunction
Just before a new Request is constructed and enqueued to the RequestQueue, this function can be used
to remove it or modify its contents such as userData
, payload
or, most importantly uniqueKey
. This is useful
when you need to enqueue multiple Requests
to the queue that share the same URL, but differ in methods or payloads,
or to dynamically update or create userData
.
For example: by adding keepUrlFragment: true
to the request
object, URL fragments will not be removed
when uniqueKey
is computed.
Example:
{
transformRequestFunction: (request) => {
request.userData.foo = 'bar';
request.keepUrlFragment = true;
return request;
}
}
Note that transformRequestFunction
has a priority over request options
specified in globs
, regexps
, or pseudoUrls
objects,
and thus some options could be over-written by transformRequestFunction
.
optionalurls
An array of URLs to enqueue.
optionaluserData
Sets Request.userData for newly enqueued requests.
A base URL that will be used to resolve relative URLs when using Cheerio. Ignored when using Puppeteer, since the relative URL resolution is done inside the browser automatically.