Version: Next

AddRequestsKwargs

Keyword arguments for the add_requests methods.

Hierarchy

EnqueueLinksKwargs
- AddRequestsKwargs

Index

Properties

base_url

base_url: NotRequired[str]

Base URL to be used for relative URLs.

exclude

exclude: NotRequired[Sequence[re.Pattern | Glob]]

List of regular expressions or globs that URLs must not match to be enqueued.

include

include: NotRequired[Sequence[re.Pattern | Glob]]

List of regular expressions or globs that URLs must match to be enqueued.

limit

limit: NotRequired[int]

Maximum number of requests to be enqueued.

requests

requests: Sequence[str | Request]

Requests to be added to the RequestManager.

rq_alias

rq_alias: str | None

Alias of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

rq_id

rq_id: str | None

ID of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

rq_name

rq_name: str | None

Name of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

strategy

strategy: NotRequired[EnqueueStrategy]

Enqueue strategy that determines which URLs are enqueued based on their relation to the page the crawler is currently on. With enqueue_links it filters the links extracted from the page. With add_requests it filters the explicitly provided requests.

Options: all: Enqueue every URL encountered, regardless of the target domain. Use this option to ensure that all URLs, including those leading to external websites, are followed. same-domain: Enqueue URLs that share the same domain name as the current page, including any subdomains. This strategy is ideal for crawling within the same top-level domain while still allowing for subdomain exploration. same-hostname: Enqueue URLs only if they match the exact hostname of the current page. This is the default for enqueue_links and restricts the crawl to the current hostname, excluding subdomains. same-origin: Enqueue URLs that share the same origin as the current page. The origin is defined by the combination of protocol, domain, and port, ensuring a strict scope for the crawl.