@crawlee/utils
Index
Namespaces
Classes
Interfaces
Type Aliases
Variables
Functions
Type Aliases
CheerioRoot
Variables
constCLOUDFLARE_RETRY_CSS_SELECTORS
constRETRY_CSS_SELECTORS
constROTATE_PROXY_ERRORS
Content of proxy errors that should trigger a retry, as the proxy is likely getting blocked / is malfunctioning.
constURL_NO_COMMAS_REGEX
Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
constURL_WITH_COMMAS_REGEX
Regular expression that, in addition to the default regular expression URL_NO_COMMAS_REGEX, supports matching commas in URL path and query.
Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
CSS selectors for elements that should trigger a retry, as the crawler is likely getting blocked.