Skip to main content
Version: Next

ParseSitemapOptions

Index

Properties

Properties

optionalemitNestedSitemaps

emitNestedSitemaps?: boolean

If set to true, elements referring to other sitemaps will be emitted as special objects with originSitemapUrl set to null.

optionalenqueueStrategy

enqueueStrategy?: EnqueueStrategy | all | same-domain | same-hostname | same-origin = EnqueueStrategy | all | same-domain | same-hostname | same-origin

Keep only sitemap-derived URLs (nested <sitemap> and <url> entries) matching this strategy relative to the parent sitemap URL; non-http(s) schemes are always dropped. Skipped for raw string sources (no parent URL). Pass 'all' to disable host filtering.

optionalhttpClient

httpClient?: BaseHttpClient

Custom HTTP client to be used for fetching sitemaps.

optionallogger

logger?: CrawleeLogger

Optional logger for reporting warnings during sitemap parsing.

optionalmaxDepth

maxDepth?: number

Maximum depth of nested sitemaps to follow.

optionalnestedSitemapFilter

nestedSitemapFilter?: (sitemapUrl) => boolean

Optional filter for nested sitemap URLs discovered in sitemap index files. Called with the URL of each child sitemap before it is fetched. Return true to include the sitemap, false to skip it. If not provided, all nested sitemaps are followed.

Type declaration

    • (sitemapUrl): boolean

    • Parameters

      • sitemapUrl: string

      Returns boolean

optionalreportNetworkErrors

reportNetworkErrors?: boolean = true

If true, the parser will log a warning if it fails to fetch a sitemap due to a network error

optionalsitemapRetries

sitemapRetries?: number

Number of retries for fetching sitemaps. The counter resets for each nested sitemap.

optionaltimeoutMillis

timeoutMillis?: number

Timeout settings for network requests when fetching sitemaps. By default this is 30000 milliseconds (30 seconds).

Page Options