Skip to main content

RedisDatasetClient

Redis implementation of the dataset client.

This client persists dataset items to Redis using JSON arrays for efficient storage and retrieval. Items are stored as JSON objects with automatic ordering preservation through Redis list operations.

The dataset data is stored in Redis using the following key pattern:

  • datasets:{name}:items - Redis JSON array containing all dataset items.
  • datasets:{name}:metadata - Redis JSON object containing dataset metadata.

Items must be JSON-serializable dictionaries. Single items or lists of items can be pushed to the dataset. The item ordering is preserved through Redis JSON array operations. All operations provide atomic consistency through Redis transactions and pipeline operations.

Hierarchy

Index

Methods

Properties

Methods

__init__

  • __init__(storage_name, storage_id, redis): None

  • Initialize a new instance.

    Preferably use the RedisDatasetClient.open class method to create a new instance.

    Parameters

    • storage_name: str

      Internal storage name used for Redis keys.

    • storage_id: str

      Unique identifier for the dataset.

    • redis: Redis

      Redis client instance.

    Returns None

drop

  • async drop(): None

  • Drop the whole dataset and remove all its items.

    The backend method for the Dataset.drop call.

    Returns None

get_data

  • async get_data(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): DatasetItemsListPage

  • Get data from the dataset with various filtering options.

    The backend method for the Dataset.get_data call.

    Parameters

    • optionalkeyword-onlyoffset: int = 0
    • optionalkeyword-onlylimit: int | None = 999_999_999_999
    • optionalkeyword-onlyclean: bool = False
    • optionalkeyword-onlydesc: bool = False
    • optionalkeyword-onlyfields: list[str] | None = None
    • optionalkeyword-onlyomit: list[str] | None = None
    • optionalkeyword-onlyunwind: list[str] | None = None
    • optionalkeyword-onlyskip_empty: bool = False
    • optionalkeyword-onlyskip_hidden: bool = False
    • optionalkeyword-onlyflatten: list[str] | None = None
    • optionalkeyword-onlyview: str | None = None

    Returns DatasetItemsListPage

get_metadata

iterate_items

  • async iterate_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): AsyncIterator[dict[str, Any]]

  • Iterate over dataset items one by one.

    This method yields items individually instead of loading all items at once, which is more memory efficient for large datasets.

    Parameters

    • optionalkeyword-onlyoffset: int = 0
    • optionalkeyword-onlylimit: int | None = None
    • optionalkeyword-onlyclean: bool = False
    • optionalkeyword-onlydesc: bool = False
    • optionalkeyword-onlyfields: list[str] | None = None
    • optionalkeyword-onlyomit: list[str] | None = None
    • optionalkeyword-onlyunwind: list[str] | None = None
    • optionalkeyword-onlyskip_empty: bool = False
    • optionalkeyword-onlyskip_hidden: bool = False

    Returns AsyncIterator[dict[str, Any]]

open

  • Open or create a new Redis dataset client.

    This method attempts to open an existing dataset from the Redis database. If a dataset with the specified ID or name exists, it loads the metadata from the database. If no existing store is found, a new one is created.

    Parameters

    • keyword-onlyid: str | None

      The ID of the dataset. If not provided, a random ID will be generated.

    • keyword-onlyname: str | None

      The name of the dataset for named (global scope) storages.

    • keyword-onlyalias: str | None

      The alias of the dataset for unnamed (run scope) storages.

    • keyword-onlyredis: Redis

      Redis client instance.

    Returns RedisDatasetClient

purge

  • async purge(): None

  • Purge all items from the dataset.

    The backend method for the Dataset.purge call.

    Returns None

push_data

  • async push_data(data): None

  • Push data to the dataset.

    The backend method for the Dataset.push_data call.

    Parameters

    • data: list[Any] | dict[str, Any]

    Returns None

Properties

metadata_key

metadata_key: str

Return the Redis key for the metadata of this storage.

redis

redis: Redis

Return the Redis client instance.

Page Options