RequestOptions <UserData>

Specifies required and optional fields for constructing a Request.

Index

Properties

• headers
• keepUrlFragment
• label
• maxRetries
• method
• noRetry
• payload
• skipNavigation
• uniqueKey
• url
• useExtendedUniqueKey
• userData

Properties

optionalheaders

headers?: Record<string, string>

HTTP headers in the following format:

{
    Accept: 'text/html',
    'Content-Type': 'application/json'
}

optionalkeepUrlFragment

keepUrlFragment?: boolean = false

If false then the hash part of a URL is removed when computing the uniqueKey property. For example, this causes the http://www.example.com#foo and http://www.example.com#bar URLs to have the same uniqueKey of http://www.example.com and thus the URLs are considered equal. Note that this option only has an effect if uniqueKey is not set.

optionallabel

label?: string

Shortcut for setting userData: { label: '...' }.

optionalmaxRetries

maxRetries?: number

Maximum number of retries for this request. Allows to override the global maxRequestRetries option of BasicCrawler.

optionalmethod

method?: get | options | AllowedHttpMethods | head | post | put | delete | trace | connect | patch = get | options | AllowedHttpMethods | head | post | put | delete | trace | connect | patch

optionalnoRetry

noRetry?: boolean = false

The true value indicates that the request will not be automatically retried on error.

optionalpayload

payload?: string

HTTP request payload, e.g. for POST requests.

optionalskipNavigation

skipNavigation?: boolean = false

If set to true then the crawler processing this request evaluates the requestHandler immediately without prior browser navigation.

optionaluniqueKey

uniqueKey?: string

A unique key identifying the request. Two requests with the same uniqueKey are considered as pointing to the same URL.

If uniqueKey is not provided, then it is automatically generated by normalizing the URL. For example, the URL of HTTP://www.EXAMPLE.com/something/ will produce the uniqueKey of http://www.example.com/something.

The keepUrlFragment option determines whether URL hash fragment is included in the uniqueKey or not.

The useExtendedUniqueKey options determines whether method and payload are included in the uniqueKey, producing a uniqueKey in the following format: METHOD(payloadHash):normalizedUrl. This is useful when requests point to the same URL, but with different methods and payloads. For example: form submits.

Pass an arbitrary non-empty text value to the uniqueKey property to override the default behavior and specify which URLs shall be considered equal.

url

url: string

URL of the web page to crawl. It must be a non-empty string.

optionaluseExtendedUniqueKey

useExtendedUniqueKey?: boolean = false

If true then the uniqueKey is computed not only from the URL, but also from the method and payload properties. This is useful when making requests to the same URL that are differentiated by method or payload, such as form submit navigations in browsers.

optionaluserData

userData?: UserData

Custom user data assigned to the request. Use this to save any request related data to the request's scope, keeping them accessible on retries, failures etc.