Skip to main content

StorageClient

Base class for storage clients.

The StorageClient serves as an abstract base class that defines the interface for accessing Crawlee's storage types: datasets, key-value stores, and request queues. It provides methods to open clients for each of these storage types and handles common functionality.

Storage clients implementations can be provided for various backends (file system, memory, databases, various cloud providers, etc.) to support different use cases from development to production environments.

Each storage client implementation is responsible for ensuring proper initialization, data persistence (where applicable), and consistent access patterns across all storage types it supports.

Hierarchy

Index

Methods

create_dataset_client

  • async create_dataset_client(*, id, name, configuration): DatasetClient

create_kvs_client

create_rq_client

get_rate_limit_errors

  • get_rate_limit_errors(): dict[int, int]