StorageClient

Base class for storage clients.

The StorageClient serves as an abstract base class that defines the interface for accessing Crawlee's storage types: datasets, key-value stores, and request queues. It provides methods to open clients for each of these storage types and handles common functionality.

Storage clients implementations can be provided for various backends (file system, memory, databases, various cloud providers, etc.) to support different use cases from development to production environments.

Each storage client implementation is responsible for ensuring proper initialization, data persistence (where applicable), and consistent access patterns across all storage types it supports.

Hierarchy

• StorageClient
  • MemoryStorageClient
  • FileSystemStorageClient

Index

Methods

• create_dataset_client
• create_kvs_client
• create_rq_client
• get_rate_limit_errors

Methods

create_dataset_client

• async create_dataset_client(*, id, name, configuration): DatasetClient

• Overrides StorageClient.create_dataset_client

  Create a dataset client.

  ---

  Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyconfiguration: Configuration | None = None

  Returns DatasetClient

create_kvs_client

• async create_kvs_client(*, id, name, configuration): KeyValueStoreClient

• Overrides StorageClient.create_kvs_client

  Create a key-value store client.

  ---

  Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyconfiguration: Configuration | None = None

  Returns KeyValueStoreClient

create_rq_client

• async create_rq_client(*, id, name, configuration): RequestQueueClient

• Overrides StorageClient.create_rq_client

  Create a request queue client.

  ---

  Parameters

  • optionalkeyword-onlyid: str | None = None

  • optionalkeyword-onlyname: str | None = None

  • optionalkeyword-onlyconfiguration: Configuration | None = None

  Returns RequestQueueClient

get_rate_limit_errors

• get_rate_limit_errors(): dict[int, int]

• Inherited from StorageClient.get_rate_limit_errors

  Return statistics about rate limit errors encountered by the HTTP client in storage client.

  ---

  Returns dict[int, int]