Sitemap

Loads one or more sitemaps from given URLs, following references in sitemap index files, and exposes the contained URLs.

Example usage:

// Load a sitemap
const sitemap = await Sitemap.load(['https://example.com/sitemap.xml', 'https://example.com/sitemap_2.xml.gz']);

// Enqueue all the contained URLs (including those from sub-sitemaps from sitemap indexes)
await crawler.addRequests(sitemap.urls);

Index

Constructors

• constructor

Properties

• urls

Methods

• load

Constructors

constructor

• new Sitemap(urls: string[]): Sitemap

• Parameters

  • urls: string[]

  Returns Sitemap

Properties

readonlyurls

urls: string[]

Methods

staticload

• load(urls: string | string[], proxyUrl?: string): Promise<Sitemap>

• Fetch sitemap content from given URL or URLs and return URLs of referenced pages.

  ---

  Parameters

  • urls: string | string[]

    sitemap URL(s)

  • optionalproxyUrl: string

    URL of a proxy to be used for fetching sitemap contents

  Returns Promise<Sitemap>