Skip to main content

BaseRequestQueueClient

crawlee.base_storage_client.base_request_queue_client.BaseRequestQueueClient

Abstract base class for request queue resource clients.

These clients are specific to the type of resource they manage and operate under a designated storage client, like a memory storage client.

Index

Methods

add_request

  • async add_request(request, *, forefront): ProcessedRequest
  • Add a request to the queue.


    Parameters

    • request: Request
    • forefront: bool = Falsekeyword-only

    Returns ProcessedRequest

batch_add_requests

  • async batch_add_requests(requests, *, forefront): BatchRequestsOperationResponse
  • Add a batch of requests to the queue.


    Parameters

    • requests: Sequence[Request]
    • forefront: bool = Falsekeyword-only

    Returns BatchRequestsOperationResponse

batch_delete_requests

  • async batch_delete_requests(requests): BatchRequestsOperationResponse
  • Delete given requests from the queue.


    Parameters

    • requests: list[Request]

    Returns BatchRequestsOperationResponse

delete

  • async delete(): None
  • Permanently delete the request queue managed by this client.


    Returns None

delete_request

  • async delete_request(request_id): None
  • Delete a request from the queue.


    Parameters

    • request_id: str

    Returns None

delete_request_lock

  • async delete_request_lock(request_id, *, forefront): None
  • Delete the lock on a specific request in the queue.


    Parameters

    • request_id: str
    • forefront: bool = Falsekeyword-only

    Returns None

get

  • async get(): RequestQueueMetadata | None
  • Get metadata about the request queue being managed by this client.


    Returns RequestQueueMetadata | None

get_request

  • async get_request(request_id): Request | None
  • Retrieve a request from the queue.


    Parameters

    • request_id: str

    Returns Request | None

list_and_lock_head

  • async list_and_lock_head(*, lock_secs, limit): RequestQueueHeadWithLocks
  • Fetch and lock a specified number of requests from the start of the queue.

    Retrieves and locks the first few requests of a queue for the specified duration. This prevents the requests from being fetched by another client until the lock expires.


    Parameters

    • lock_secs: intkeyword-only
    • limit: int | None = Nonekeyword-only

    Returns RequestQueueHeadWithLocks

list_head

  • async list_head(*, limit): RequestQueueHead
  • Retrieve a given number of requests from the beginning of the queue.


    Parameters

    • limit: int | None = Nonekeyword-only

    Returns RequestQueueHead

list_requests

  • async list_requests(*, limit, exclusive_start_id): RequestListResponse
  • List requests from the queue.


    Parameters

    • limit: int | None = Nonekeyword-only
    • exclusive_start_id: str | None = Nonekeyword-only

    Returns RequestListResponse

prolong_request_lock

  • async prolong_request_lock(request_id, *, forefront, lock_secs): ProlongRequestLockResponse
  • Prolong the lock on a specific request in the queue.


    Parameters

    • request_id: str
    • forefront: bool = Falsekeyword-only
    • lock_secs: intkeyword-only

    Returns ProlongRequestLockResponse

update

  • async update(*, name): RequestQueueMetadata
  • Update the request queue metadata.


    Parameters

    • name: str | None = Nonekeyword-only

    Returns RequestQueueMetadata

update_request

  • async update_request(request, *, forefront): ProcessedRequest
  • Update a request in the queue.


    Parameters

    • request: Request
    • forefront: bool = Falsekeyword-only

    Returns ProcessedRequest