Skip to main content

RequestList

Represents a (potentially very large) list of URLs to crawl.

Disclaimer: The RequestList class is in its early version and is not fully implemented. It is currently intended mainly for testing purposes and small-scale projects. The current implementation is only in-memory storage and is very limited. It will be (re)implemented in the future. For more details, see the GitHub issue: https://github.com/apify/crawlee-python/issues/99. For production usage we recommend to use the RequestQueue.

Hierarchy

Index

Methods

__init__

  • __init__(*, requests, name): None
  • A default constructor.


    Parameters

    • optionalkeyword-onlyrequests: (Iterable[str | Request] | AsyncIterable[str | Request]) | None = None

      The request objects (or their string representations) to be added to the provider.

    • optionalkeyword-onlyname: str | None = None

      A name of the request list.

    Returns None

fetch_next_request

  • async fetch_next_request(): Request | None
  • Returns the next request to be processed, or null if there are no more pending requests.


    Returns Request | None

get_handled_count

  • async get_handled_count(): int
  • Returns the number of handled requests.


    Returns int

get_total_count

  • async get_total_count(): int
  • Returns an offline approximation of the total number of requests in the source (i.e. pending + handled).


    Returns int

is_empty

  • async is_empty(): bool
  • Returns True if there are no more requests in the source (there might still be unfinished requests).


    Returns bool

is_finished

  • async is_finished(): bool
  • Returns True if all requests have been handled.


    Returns bool

mark_request_as_handled

  • Marks a request as handled after a successful processing (or after giving up retrying).


    Parameters

    • optionalkeyword-onlyrequest: Request

    Returns ProcessedRequest | None

to_tandem

  • Combine the loader with a request manager to support adding and reclaiming requests.


    Parameters

    • optionalkeyword-onlyrequest_manager: RequestManager | None = None

      Request manager to combine the loader with. If None is given, the default request queue is used.

    Returns RequestManagerTandem

Properties

name

name: str | None