EnqueueLinksKwargs

Keyword arguments for the enqueue_links methods.

Hierarchy

• EnqueueLinksKwargs
  • AddRequestsKwargs

Index

Properties

• base_url
• exclude
• include
• limit
• strategy

Properties

keyword-onlyoptionalbase_url

base_url: str

Base URL to be used for relative URLs.

keyword-onlyoptionalexclude

exclude: list[re.Pattern | Glob]

List of regular expressions or globs that URLs must not match to be enqueued.

keyword-onlyoptionalinclude

include: list[re.Pattern | Glob]

List of regular expressions or globs that URLs must match to be enqueued.

keyword-onlyoptionallimit

limit: int

Maximum number of requests to be enqueued.

keyword-onlyoptionalstrategy

strategy: Literal['all', 'same-domain', 'same-hostname', 'same-origin']

Enqueue strategy to be used for determining which links to extract and enqueue.

Options: all: Enqueue every link encountered, regardless of the target domain. Use this option to ensure that all links, including those leading to external websites, are followed. same-domain: Enqueue links that share the same domain name as the current page, including any subdomains. This strategy is ideal for crawling within the same top-level domain while still allowing for subdomain exploration. same-hostname: Enqueue links only if they match the exact hostname of the current page. This is the default behavior and restricts the crawl to the current hostname, excluding subdomains. same-origin: Enqueue links that share the same origin as the current page. The origin is defined by the combination of protocol, domain, and port, ensuring a strict scope for the crawl.