RequestList

Represents a (potentially very large) list of URLs to crawl.

Disclaimer: The RequestList class is in its early version and is not fully implemented. It is currently intended mainly for testing purposes and small-scale projects. The current implementation is only in-memory storage and is very limited. It will be (re)implemented in the future. For more details, see the GitHub issue: https://github.com/apify/crawlee-python/issues/99. For production usage we recommend to use the RequestQueue.

Hierarchy

• RequestLoader
  • RequestList

Index

Methods

• __init__
• fetch_next_request
• get_handled_count
• get_total_count
• is_empty
• is_finished
• mark_request_as_handled
• to_tandem

Properties

• name

Methods

__init__

• __init__(*, requests, name): None

• A default constructor.

  ---

  Parameters

  • optionalkeyword-onlyrequests: (Iterable[str | Request] | AsyncIterable[str | Request]) | None = None

    The request objects (or their string representations) to be added to the provider.

  • optionalkeyword-onlyname: str | None = None

    A name of the request list.

  Returns None

fetch_next_request

• async fetch_next_request(): Request | None

• Returns the next request to be processed, or null if there are no more pending requests.

  ---

  Returns Request | None

get_handled_count

• async get_handled_count(): int

• Returns the number of handled requests.

  ---

  Returns int

get_total_count

• async get_total_count(): int

• Returns an offline approximation of the total number of requests in the source (i.e. pending + handled).

  ---

  Returns int

is_empty

• async is_empty(): bool

• Returns True if there are no more requests in the source (there might still be unfinished requests).

  ---

  Returns bool

is_finished

• async is_finished(): bool

• Returns True if all requests have been handled.

  ---

  Returns bool

mark_request_as_handled

• async mark_request_as_handled(*, request): ProcessedRequest | None

• Marks a request as handled after a successful processing (or after giving up retrying).

  ---

  Parameters

  • optionalkeyword-onlyrequest: Request

  Returns ProcessedRequest | None

to_tandem

• async to_tandem(*, request_manager): RequestManagerTandem

• Combine the loader with a request manager to support adding and reclaiming requests.

  ---

  Parameters

  • optionalkeyword-onlyrequest_manager: RequestManager | None = None

    Request manager to combine the loader with. If None is given, the default request queue is used.

  Returns RequestManagerTandem

Properties

name

name: str | None