FileSystemKeyValueStoreClient

File system implementation of the key-value store client.

This client persists data to the file system, making it suitable for scenarios where data needs to survive process restarts. Keys are mapped to file paths in a directory structure following the pattern:

{STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}

Binary data is stored as-is, while JSON and text data are stored in human-readable format. The implementation automatically handles serialization based on the content type and maintains metadata about each record.

This implementation is ideal for long-running crawlers where persistence is important and for development environments where you want to easily inspect the stored data between runs.

Hierarchy

• KeyValueStoreClient
  • FileSystemKeyValueStoreClient

Index

Methods

• __init__
• delete_value
• drop
• get_metadata
• get_public_url
• get_value
• iterate_keys
• open
• purge
• record_exists
• set_value

Properties

• path_to_kvs
• path_to_metadata

Methods

__init__

• __init__(*, metadata, storage_dir, lock): None

• Initialize a new instance.

  Preferably use the FileSystemKeyValueStoreClient.open class method to create a new instance.

  ---

  Parameters

  • keyword-onlymetadata: KeyValueStoreMetadata

  • keyword-onlystorage_dir: Path

  • keyword-onlylock: asyncio.Lock

  Returns None

delete_value

• async delete_value(*, key): None

• Overrides KeyValueStoreClient.delete_value

  Delete a value from the key-value store by its key.

  The backend method for the KeyValueStore.delete_value call.

  ---

  Parameters

  • keyword-onlykey: str

  Returns None

drop

• async drop(): None

• Overrides KeyValueStoreClient.drop

  Drop the whole key-value store and remove all its values.

  The backend method for the KeyValueStore.drop call.

  ---

  Returns None

get_metadata

• async get_metadata(): KeyValueStoreMetadata

• Overrides KeyValueStoreClient.get_metadata

  Get the metadata of the key-value store.

  ---

  Returns KeyValueStoreMetadata

get_public_url

• async get_public_url(*, key): str

• Overrides KeyValueStoreClient.get_public_url

  Return a file:// URL for the given key.

  ---

  Parameters

  • keyword-onlykey: str

    The key to get the public URL for.

  Returns str

get_value

• async get_value(*, key): KeyValueStoreRecord | None

• Overrides KeyValueStoreClient.get_value

  Retrieve the given record from the key-value store.

  The backend method for the KeyValueStore.get_value call.

  ---

  Parameters

  • keyword-onlykey: str

  Returns KeyValueStoreRecord | None

iterate_keys

• async iterate_keys(*, exclusive_start_key, limit): AsyncIterator[KeyValueStoreRecordMetadata]

• Overrides KeyValueStoreClient.iterate_keys

  Iterate over all the existing keys in the key-value store.

  The backend method for the KeyValueStore.iterate_keys call.

  ---

  Parameters

  • optionalkeyword-onlyexclusive_start_key: str | None = None

  • optionalkeyword-onlylimit: int | None = None

  Returns AsyncIterator[KeyValueStoreRecordMetadata]

open

• async open(*, id, name, configuration): FileSystemKeyValueStoreClient

• Open or create a file system key-value store client.

  This method attempts to open an existing key-value store from the file system. If a KVS with the specified ID or name exists, it loads the metadata from the stored files. If no existing store is found, a new one is created.

  ---

  Parameters

  • keyword-onlyid: str | None

    The ID of the key-value store to open. If provided, searches for existing store by ID.

  • keyword-onlyname: str | None

    The name of the key-value store to open. If not provided, uses the default store.

  • keyword-onlyconfiguration: Configuration

    The configuration object containing storage directory settings.

  Returns FileSystemKeyValueStoreClient

purge

• async purge(): None

• Overrides KeyValueStoreClient.purge

  Purge all items from the key-value store.

  The backend method for the KeyValueStore.purge call.

  ---

  Returns None

record_exists

• async record_exists(*, key): bool

• Overrides KeyValueStoreClient.record_exists

  Check if a record with the given key exists in the key-value store.

  ---

  Parameters

  • keyword-onlykey: str

    The key to check for existence.

  Returns bool

set_value

• async set_value(*, key, value, content_type): None

• Overrides KeyValueStoreClient.set_value

  Set a value in the key-value store by its key.

  The backend method for the KeyValueStore.set_value call.

  ---

  Parameters

  • keyword-onlykey: str

  • keyword-onlyvalue: Any

  • optionalkeyword-onlycontent_type: str | None = None

  Returns None

Properties

path_to_kvs

path_to_kvs: Path

The full path to the key-value store directory.

path_to_metadata

path_to_metadata: Path

The full path to the key-value store metadata file.