RequestOptions <UserData>
Index
Properties
optionalheaders
HTTP headers in the following format:
{
Accept: 'text/html',
'Content-Type': 'application/json'
}
optionalkeepUrlFragment
If false
then the hash part of a URL is removed when computing the uniqueKey
property.
For example, this causes the http://www.example.com#foo
and http://www.example.com#bar
URLs
to have the same uniqueKey
of http://www.example.com
and thus the URLs are considered equal.
Note that this option only has an effect if uniqueKey
is not set.
optionallabel
Shortcut for setting userData: { label: '...' }
.
optionalmaxRetries
Maximum number of retries for this request. Allows to override the global maxRequestRetries
option of BasicCrawler
.
optionalmethod
optionalnoRetry
The true
value indicates that the request will not be automatically retried on error.
optionalpayload
HTTP request payload, e.g. for POST requests.
optionalskipNavigation
If set to true
then the crawler processing this request evaluates
the requestHandler
immediately without prior browser navigation.
optionaluniqueKey
A unique key identifying the request.
Two requests with the same uniqueKey
are considered as pointing to the same URL.
If uniqueKey
is not provided, then it is automatically generated by normalizing the URL.
For example, the URL of HTTP://www.EXAMPLE.com/something/
will produce the uniqueKey
of http://www.example.com/something
.
The keepUrlFragment
option determines whether URL hash fragment is included in the uniqueKey
or not.
The useExtendedUniqueKey
options determines whether method and payload are included in the uniqueKey
,
producing a uniqueKey
in the following format: METHOD(payloadHash):normalizedUrl
. This is useful
when requests point to the same URL, but with different methods and payloads. For example: form submits.
Pass an arbitrary non-empty text value to the uniqueKey
property
to override the default behavior and specify which URLs shall be considered equal.
url
URL of the web page to crawl. It must be a non-empty string.
optionaluseExtendedUniqueKey
If true
then the uniqueKey
is computed not only from the URL, but also from the method and payload
properties. This is useful when making requests to the same URL that are differentiated by method
or payload, such as form submit navigations in browsers.
optionaluserData
Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries, failures etc.
Specifies required and optional fields for constructing a Request.