Skip to main content
Version: Next

SitemapRequestLoaderOptions

Hierarchy

  • UrlConstraints
    • SitemapRequestLoaderOptions

Index

Properties

Properties

optionalenqueueStrategy

enqueueStrategy?: EnqueueStrategy | all | same-domain | same-hostname | same-origin = EnqueueStrategy | all | same-domain | same-hostname | same-origin

Keep only sitemap-derived URLs matching this strategy relative to the parent sitemap URL; non-http(s) schemes are always dropped. The filtering stays enforced after navigation (e.g. across redirects). Pass 'all' to disable host filtering.

optionalinheritedexclude

exclude?: readonly UrlPatternInput[]

An array of URL patterns. Matching URLs will not be included.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

optionalhttpClient

httpClient?: BaseHttpClient

Custom HTTP client to be used for sitemap loading.

optionalinheritedinclude

include?: readonly UrlPatternInput[]

An array of URL patterns that URLs must match to be included.

Accepts glob pattern strings, { glob: string } objects, RegExp instances, or { regexp: RegExp } objects.

Glob matching is always case-insensitive. If you need case-sensitive matching, use a RegExp.

If include is an empty array or undefined, then the SitemapRequestLoader includes all the URLs from the sitemap.

optionalmaxBufferSize

maxBufferSize?: number = 200

Maximum number of buffered URLs for the sitemap loading stream. If the buffer is full, the stream will pause until the buffer is drained.

optionalparseSitemapOptions

parseSitemapOptions?: Omit<ParseSitemapOptions, emitNestedSitemaps | maxDepth>

Advanced options for the underlying parseSitemap call.

optionalpersistenceOptions

persistenceOptions?: { enable?: boolean }

Persistence-related options to control how and when crawler's data gets persisted.

Type declaration

  • optionalenable?: boolean

    Use this flag to disable or enable periodic persistence to key value store.

optionalpersistStateKey

persistStateKey?: string

Key for persisting the state of the request list in the KeyValueStore.

optionalproxyUrl

proxyUrl?: string

Proxy URL to be used for sitemap loading.

optionalsignal

signal?: AbortSignal

Abort signal to be used for sitemap loading.

sitemapUrls

sitemapUrls: string[]

List of sitemap URLs to parse.

optionaltimeoutMillis

timeoutMillis?: number

Timeout for sitemap loading in milliseconds. If both signal and timeoutMillis are provided, either of them can abort the loading.

Page Options