Skip to main content

MemoryStorageClient

Memory implementation of the storage client.

This storage client provides access to datasets, key-value stores, and request queues that store all data in memory using Python data structures (lists and dictionaries). No data is persisted between process runs, meaning all stored data is lost when the program terminates.

The memory implementation provides fast access to data but is limited by available memory and does not support data sharing across different processes. All storage operations happen entirely in memory with no disk operations.

The memory storage client is useful for testing and development environments, or short-lived crawler operations where persistence is not required.

Hierarchy

Index

Methods

create_dataset_client

  • async create_dataset_client(*, id, name, configuration): DatasetClient

create_kvs_client

create_rq_client

get_rate_limit_errors

  • get_rate_limit_errors(): dict[int, int]