@crawlee/utils

Index

References

• ArgumentValidationError
• EnqueueStrategy

Namespaces

• social

Classes

• RobotsTxtFile
• Sitemap

Interfaces

• DownloadListOfUrlsOptions
• ExtractUrlsOptions
• MicrodataItem
• OpenGraphProperty
• ParseSitemapOptions
• RobotsTxtFileSitemapsOptions

Type Aliases

• MicrodataValue
• SitemapUrl

Functions

• discoverValidSitemaps
• downloadListOfUrls
• extractMicrodata
• extractUrls
• htmlToText
• parseOpenGraph
• parseSitemap
• sleep

References

ArgumentValidationError

Re-exports ArgumentValidationError

EnqueueStrategy

Re-exports EnqueueStrategy

Type Aliases

MicrodataValue

MicrodataValue: string | MicrodataItem

The value of a microdata property: either text or a nested item.

SitemapUrl

SitemapUrl: SitemapUrlData & { originSitemapUrl: string }