Skip to main content

MemoryStorageClient

crawlee.memory_storage_client._memory_storage_client.MemoryStorageClient

Represents an in-memory storage client for managing datasets, key-value stores, and request queues.

It emulates in-memory storage similar to the Apify platform, supporting both in-memory and local file system-based persistence.

The behavior of the storage, such as data persistence and metadata writing, can be customized via initialization parameters or environment variables.

Index

Constructors

Methods

Properties

Constructors

__init__

  • __init__(configuration): None

  • Create a new instance.

    Parameters

    • configuration: Configuration | None = None

    Returns None

Methods

add_resource_client_to_cache

  • add_resource_client_to_cache(resource_client): None

  • Add a new resource client to the internal cache.

    Parameters

    • resource_client: ResourceClient

    Returns None

dataset

  • dataset(id): DatasetClient

  • Parameters

    • id: str

    Returns DatasetClient

datasets

  • datasets(): DatasetCollectionClient

  • Returns DatasetCollectionClient

get_cached_resource_client

  • get_cached_resource_client(resource_client_class, id, name): TResourceClient | None

  • Try to return a resource client from the internal cache.

    Parameters

    • resource_client_class: type[TResourceClient]
    • id: str | None
    • name: str | None

    Returns TResourceClient | None

key_value_store

  • key_value_store(id): KeyValueStoreClient

  • Parameters

    • id: str

    Returns KeyValueStoreClient

key_value_stores

  • key_value_stores(): KeyValueStoreCollectionClient

  • Returns KeyValueStoreCollectionClient

purge_on_start

  • async purge_on_start(): None

  • Returns None

request_queue

  • request_queue(id): RequestQueueClient

  • Parameters

    • id: str

    Returns RequestQueueClient

request_queues

  • request_queues(): RequestQueueCollectionClient

  • Returns RequestQueueCollectionClient

Properties

datasets_directory

datasets_directory: str

Path to the directory containing datasets.

key_value_stores_directory

key_value_stores_directory: str

Path to the directory containing key-value stores.

persist_storage

persist_storage: bool

Whether to persist the storage.

request_queues_directory

request_queues_directory: str

Path to the directory containing request queues.

storage_dir

storage_dir: str

Path to the storage directory.

write_metadata

write_metadata: bool

Whether to write metadata to the storage.