Skip to main content

KeyValueStore

crawlee.storages.key_value_store.KeyValueStore

Represents a key-value based storage for reading data records or files.

Each record is identified by a unique key and associated with a MIME content type. This class is used within crawler runs to store inputs and outputs, typically in JSON format, but supports other types as well.

The data can be stored on a local filesystem or in the cloud, determined by the CRAWLEE_STORAGE_DIR environment variable.

By default, data is stored in {CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}, where {STORE_ID} is either "default" or specified by CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID, {KEY} is the record key, and {EXT} is the MIME type.

To open a key-value store, use the class method open, providing either an id or name along with optional config. If neither is provided, the default store for the crawler run is used. Opening a non-existent store by id raises an error, while a non-existent store by name is created.

Usage: kvs = await KeyValueStore.open(id='my_kvs_id')

Index

Constructors

__init__

  • __init__(id, name, configuration, client): None
  • Parameters

    • id: str
    • name: str | None
    • configuration: Configuration
    • client: BaseStorageClient

    Returns None

Methods

drop

  • async drop(): None
  • Returns None

get_info

  • async get_info(): KeyValueStoreMetadata | None
  • Get an object containing general information about the key value store.


    Returns KeyValueStoreMetadata | None

get_value

  • async get_value(key): Any
  • Parameters

    • key: str

    Returns Any

get_value

  • async get_value(key, default_value): T
  • Parameters

    • key: str
    • default_value: T

    Returns T

get_value

  • async get_value(key, default_value): T | None
  • Parameters

    • key: str
    • default_value: T | None = None

    Returns T | None

get_value

  • async get_value(key, default_value): T | None
  • Get a value from the KVS.


    Parameters

    • key: str
    • default_value: T | None = None

    Returns T | None

iterate_keys

  • async iterate_keys(exclusive_start_key): AsyncIterator[KeyValueStoreKeyInfo]
  • Iterate over the existing keys in the KVS.


    Parameters

    • exclusive_start_key: str | None = None

    Returns AsyncIterator[KeyValueStoreKeyInfo]

open

  • async open(*, id, name, configuration): KeyValueStore
  • Parameters

    • id: str | None = Nonekeyword-only
    • name: str | None = Nonekeyword-only
    • configuration: Configuration | None = Nonekeyword-only

    Returns KeyValueStore

set_value

  • async set_value(key, value, content_type): None
  • Set a value in the KVS.


    Parameters

    • key: str
    • value: Any
    • content_type: str | None = None

    Returns None

Properties

id

id: str

name

name: str | None