BaseStorageClient

crawlee.base_storage_client._base_storage_client.BaseStorageClient

Defines an abstract base for storage clients.

It offers interfaces to get subclients for interacting with storage resources like datasets, key-value stores, and request queues.

Index

Methods

• dataset
• datasets
• key_value_store
• key_value_stores
• purge_on_start
• request_queue
• request_queues

Methods

dataset

• dataset(id): BaseDatasetClient

• Gets a subclient for a specific dataset by its ID.

  ---

  Parameters

  • id: str

  Returns BaseDatasetClient

datasets

• datasets(): BaseDatasetCollectionClient

• Gets a subclient for dataset collection operations.

  ---

  Returns BaseDatasetCollectionClient

key_value_store

• key_value_store(id): BaseKeyValueStoreClient

• Gets a subclient for a specific key-value store by its ID.

  ---

  Parameters

  • id: str

  Returns BaseKeyValueStoreClient

key_value_stores

• key_value_stores(): BaseKeyValueStoreCollectionClient

• Gets a subclient for key-value store collection operations.

  ---

  Returns BaseKeyValueStoreCollectionClient

purge_on_start

• async purge_on_start(): None

• Performs a purge of the default storages.

  This method ensures that the purge is executed only once during the lifetime of the instance. It is primarily used to clean up residual data from previous runs to maintain a clean state. If the storage client does not support purging, leave it empty.

  ---

  Returns None

request_queue

• request_queue(id): BaseRequestQueueClient

• Gets a subclient for a specific request queue by its ID.

  ---

  Parameters

  • id: str

  Returns BaseRequestQueueClient

request_queues

• request_queues(): BaseRequestQueueCollectionClient

• Gets a subclient for request queue collection operations.

  ---

  Returns BaseRequestQueueCollectionClient