BaseRequestQueueClient

crawlee.base_storage_client._base_request_queue_client.BaseRequestQueueClient

Abstract base class for request queue resource clients.

These clients are specific to the type of resource they manage and operate under a designated storage client, like a memory storage client.

Index

Methods

• add_request
• batch_add_requests
• batch_delete_requests
• delete
• delete_request
• delete_request_lock
• get
• get_request
• list_and_lock_head
• list_head
• prolong_request_lock
• update
• update_request

Methods

add_request

• async add_request(request, *, forefront): ProcessedRequest

• Add a request to the queue.

  ---

  Parameters

  • request: Request

  • forefront: bool = Falsekeyword-only

  Returns ProcessedRequest

batch_add_requests

• async batch_add_requests(requests, *, forefront): BatchRequestsOperationResponse

• Add a batch of requests to the queue.

  ---

  Parameters

  • requests: Sequence[Request]

  • forefront: bool = Falsekeyword-only

  Returns BatchRequestsOperationResponse

batch_delete_requests

• async batch_delete_requests(requests): BatchRequestsOperationResponse

• Delete given requests from the queue.

  ---

  Parameters

  • requests: list[Request]

  Returns BatchRequestsOperationResponse

delete

• async delete(): None

• Permanently delete the request queue managed by this client.

  ---

  Returns None

delete_request

• async delete_request(request_id): None

• Delete a request from the queue.

  ---

  Parameters

  • request_id: str

  Returns None

delete_request_lock

• async delete_request_lock(request_id, *, forefront): None

• Delete the lock on a specific request in the queue.

  ---

  Parameters

  • request_id: str

  • forefront: bool = Falsekeyword-only

  Returns None

get

• async get(): RequestQueueMetadata | None

• Get metadata about the request queue being managed by this client.

  ---

  Returns RequestQueueMetadata | None

get_request

• async get_request(request_id): Request | None

• Retrieve a request from the queue.

  ---

  Parameters

  • request_id: str

  Returns Request | None

list_and_lock_head

• async list_and_lock_head(*, lock_secs, limit): RequestQueueHeadWithLocks

• Fetch and lock a specified number of requests from the start of the queue.

  Retrieves and locks the first few requests of a queue for the specified duration. This prevents the requests from being fetched by another client until the lock expires.

  ---

  Parameters

  • lock_secs: intkeyword-only

  • limit: int | None = Nonekeyword-only

  Returns RequestQueueHeadWithLocks

list_head

• async list_head(*, limit): RequestQueueHead

• Retrieve a given number of requests from the beginning of the queue.

  ---

  Parameters

  • limit: int | None = Nonekeyword-only

  Returns RequestQueueHead

prolong_request_lock

• async prolong_request_lock(request_id, *, forefront, lock_secs): ProlongRequestLockResponse

• Prolong the lock on a specific request in the queue.

  ---

  Parameters

  • request_id: str

  • forefront: bool = Falsekeyword-only

  • lock_secs: intkeyword-only

  Returns ProlongRequestLockResponse

update

• async update(*, name): RequestQueueMetadata

• Update the request queue metadata.

  ---

  Parameters

  • name: str | None = Nonekeyword-only

  Returns RequestQueueMetadata

update_request

• async update_request(request, *, forefront): ProcessedRequest

• Update a request in the queue.

  ---

  Parameters

  • request: Request

  • forefront: bool = Falsekeyword-only

  Returns ProcessedRequest