Skip to main content

MemoryStorageClient

Represents an in-memory storage client for managing datasets, key-value stores, and request queues.

It emulates in-memory storage similar to the Apify platform, supporting both in-memory and local file system-based persistence.

The behavior of the storage, such as data persistence and metadata writing, can be customized via initialization parameters or environment variables.

Hierarchy

Index

Methods

Properties

Methods

__init__

  • __init__(*, write_metadata, persist_storage, storage_dir, default_request_queue_id, default_key_value_store_id, default_dataset_id): None

  • A default constructor.

    Parameters

    • optionalkeyword-onlywrite_metadata: bool

      Whether to write metadata to the storage.

    • optionalkeyword-onlypersist_storage: bool

      Whether to persist the storage.

    • optionalkeyword-onlystorage_dir: str

      Path to the storage directory.

    • optionalkeyword-onlydefault_request_queue_id: str

      The default request queue ID.

    • optionalkeyword-onlydefault_key_value_store_id: str

      The default key-value store ID.

    • optionalkeyword-onlydefault_dataset_id: str

      The default dataset ID.

    Returns None

add_resource_client_to_cache

  • add_resource_client_to_cache(*, resource_client): None

  • Add a new resource client to the internal cache.

    Parameters

    Returns None

dataset

  • Gets a subclient for a specific dataset by its ID.

    Parameters

    • optionalkeyword-onlyid: str

    Returns BaseDatasetClient

datasets

from_config

  • Create a new instance based on the provided configuration.

    All the memory storage client parameters are taken from the configuration object.

    Parameters

    • optionalkeyword-onlyconfig: Configuration | None = None

      The configuration object.

    Returns MemoryStorageClient

get_cached_resource_client

  • get_cached_resource_client(*, resource_client_class, id, name): TResourceClient | None

  • Try to return a resource client from the internal cache.

    Parameters

    • optionalkeyword-onlyresource_client_class: type[TResourceClient]
    • optionalkeyword-onlyid: str | None
    • optionalkeyword-onlyname: str | None

    Returns TResourceClient | None

key_value_store

  • Gets a subclient for a specific key-value store by its ID.

    Parameters

    • optionalkeyword-onlyid: str

    Returns BaseKeyValueStoreClient

key_value_stores

purge_on_start

  • async purge_on_start(): None

  • Performs a purge of the default storages.

    This method ensures that the purge is executed only once during the lifetime of the instance. It is primarily used to clean up residual data from previous runs to maintain a clean state. If the storage client does not support purging, leave it empty.

    Returns None

request_queue

  • Gets a subclient for a specific request queue by its ID.

    Parameters

    • optionalkeyword-onlyid: str

    Returns BaseRequestQueueClient

request_queues

Properties

datasets_directory

datasets_directory: str

Path to the directory containing datasets.

key_value_stores_directory

key_value_stores_directory: str

Path to the directory containing key-value stores.

persist_storage

persist_storage: bool

Whether to persist the storage.

request_queues_directory

request_queues_directory: str

Path to the directory containing request queues.

storage_dir

storage_dir: str

Path to the storage directory.

write_metadata

write_metadata: bool

Whether to write metadata to the storage.

Page Options