EnqueueStrategy
Index
Enumeration Members
Enumeration Members
All
Matches any URLs found
SameDomain
Matches any URLs that have the same domain as the base URL.
For example, https://wow.an.example.com
and https://example.com
will both be matched for a base url of
https://example.com
.
This strategy will match both
http
andhttps
protocols regardless of the base URL protocol.
SameHostname
Matches any URLs that have the same hostname.
For example, https://wow.example.com/hello
will be matched for a base url of https://wow.example.com/
, but
https://example.com/hello
will not be matched.
This strategy will match both
http
andhttps
protocols regardless of the base URL protocol.
SameOrigin
Matches any URLs that have the same hostname and protocol.
For example, https://wow.example.com/hello
will be matched for a base url of https://wow.example.com/
, but
http://wow.example.com/hello
will not be matched.
This strategy will ensure the protocol of the base URL is the same as the protocol of the URL to be enqueued.
The different enqueueing strategies available.
Depending on the strategy you select, we will only check certain parts of the URLs found. Here is a diagram of each URL part and their name:
Protocol
is usuallyhttp
orhttps
Domain
represents the path without any possible subdomains to a website. For example,crawlee.dev
is the domain ofhttps://example.crawlee.dev/
Hostname
is the full path to a website, including any subdomains. For example,example.crawlee.dev
is the hostname ofhttps://example.crawlee.dev/
Origin
is the combination of theProtocol
andHostname
. For example,https://example.crawlee.dev
is the origin ofhttps://example.crawlee.dev/