MemoryStorageClient

Represents an in-memory storage client for managing datasets, key-value stores, and request queues.

It emulates in-memory storage similar to the Apify platform, supporting both in-memory and local file system-based persistence.

The behavior of the storage, such as data persistence and metadata writing, can be customized via initialization parameters or environment variables.

Hierarchy

• BaseStorageClient
  • MemoryStorageClient

Index

Methods

• __init__
• add_resource_client_to_cache
• dataset
• datasets
• get_cached_resource_client
• key_value_store
• key_value_stores
• purge_on_start
• request_queue
• request_queues

Properties

• datasets_directory
• key_value_stores_directory
• persist_storage
• request_queues_directory
• storage_dir
• write_metadata

Methods

__init__

• __init__(*, configuration): None

• A default constructor.

  ---

  Parameters

  • optionalkeyword-onlyconfiguration: Configuration | None = None

    Configuration object to use. If None, a default instance will be created.

  Returns None

add_resource_client_to_cache

• add_resource_client_to_cache(*, resource_client): None

• Add a new resource client to the internal cache.

  ---

  Parameters

  • optionalkeyword-onlyresource_client: ResourceClient

  Returns None

dataset

• dataset(*, id): BaseDatasetClient

• Gets a subclient for a specific dataset by its ID.

  ---

  Parameters

  • optionalkeyword-onlyid: str

  Returns BaseDatasetClient

datasets

• datasets(): BaseDatasetCollectionClient

• Gets a subclient for dataset collection operations.

  ---

  Returns BaseDatasetCollectionClient

get_cached_resource_client

• get_cached_resource_client(*, resource_client_class, id, name): TResourceClient | None

• Try to return a resource client from the internal cache.

  ---

  Parameters

  • optionalkeyword-onlyresource_client_class: type[TResourceClient]

  • optionalkeyword-onlyid: str | None

  • optionalkeyword-onlyname: str | None

  Returns TResourceClient | None

key_value_store

• key_value_store(*, id): BaseKeyValueStoreClient

• Gets a subclient for a specific key-value store by its ID.

  ---

  Parameters

  • optionalkeyword-onlyid: str

  Returns BaseKeyValueStoreClient

key_value_stores

• key_value_stores(): BaseKeyValueStoreCollectionClient

• Gets a subclient for key-value store collection operations.

  ---

  Returns BaseKeyValueStoreCollectionClient

purge_on_start

• async purge_on_start(): None

• Performs a purge of the default storages.

  This method ensures that the purge is executed only once during the lifetime of the instance. It is primarily used to clean up residual data from previous runs to maintain a clean state. If the storage client does not support purging, leave it empty.

  ---

  Returns None

request_queue

• request_queue(*, id): BaseRequestQueueClient

• Gets a subclient for a specific request queue by its ID.

  ---

  Parameters

  • optionalkeyword-onlyid: str

  Returns BaseRequestQueueClient

request_queues

• request_queues(): BaseRequestQueueCollectionClient

• Gets a subclient for request queue collection operations.

  ---

  Returns BaseRequestQueueCollectionClient

Properties

datasets_directory

datasets_directory: str

Path to the directory containing datasets.

key_value_stores_directory

key_value_stores_directory: str

Path to the directory containing key-value stores.

persist_storage

persist_storage: bool

Whether to persist the storage.

request_queues_directory

request_queues_directory: str

Path to the directory containing request queues.

storage_dir

storage_dir: str

Path to the storage directory.

write_metadata

write_metadata: bool

Whether to write metadata to the storage.