AddRequestsFunction

Function for adding requests to the RequestManager, with optional filtering.

It simplifies the process of adding requests to the RequestManager. It automatically opens the specified one and adds the provided requests.

Index

Methods

• __call__

Methods

__call__

• __call__(requests, rq_id, rq_name, rq_alias, *, limit, base_url, strategy, include, exclude): Coroutine[None, None, None]

• Call dunder method.

  ---

  Parameters

  • requests: Sequence[str | Request]

    Requests to be added to the RequestManager.

  • optionalrq_id: str | None = None

    ID of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

  • optionalrq_name: str | None = None

    Name of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

  • optionalrq_alias: str | None = None

    Alias of the RequestQueue to add the requests to. Only one of rq_id, rq_name or rq_alias can be provided.

  • keyword-onlyoptionallimit: int

    Maximum number of requests to be enqueued.

  • keyword-onlyoptionalbase_url: str

    Base URL to be used for relative URLs.

  • keyword-onlyoptionalstrategy: EnqueueStrategy

    Enqueue strategy that determines which URLs are enqueued based on their relation to the page the crawler is currently on. With enqueue_links it filters the links extracted from the page. With add_requests it filters the explicitly provided requests.

    Options: all: Enqueue every URL encountered, regardless of the target domain. Use this option to ensure that all URLs, including those leading to external websites, are followed. same-domain: Enqueue URLs that share the same domain name as the current page, including any subdomains. This strategy is ideal for crawling within the same top-level domain while still allowing for subdomain exploration. same-hostname: Enqueue URLs only if they match the exact hostname of the current page. This is the default for enqueue_links and restricts the crawl to the current hostname, excluding subdomains. same-origin: Enqueue URLs that share the same origin as the current page. The origin is defined by the combination of protocol, domain, and port, ensuring a strict scope for the crawl.

  • keyword-onlyoptionalinclude: Sequence[re.Pattern | Glob]

    List of regular expressions or globs that URLs must match to be enqueued.

  • keyword-onlyoptionalexclude: Sequence[re.Pattern | Glob]

    List of regular expressions or globs that URLs must not match to be enqueued.

  Returns Coroutine[None, None, None]