Skip to main content

RedisStorageClient

Redis implementation of the storage client.

This storage client provides access to datasets, key-value stores, and request queues that persist data to a Redis database v8.0+. Each storage type uses Redis-specific data structures and key patterns for efficient storage and retrieval.

The client accepts either a Redis connection string or a pre-configured Redis client instance. Exactly one of these parameters must be provided during initialization.

Storage types use the following Redis data structures:

  • Datasets: Redis JSON arrays for item storage with metadata in JSON objects
  • Key-value stores: Redis hashes for key-value pairs with separate metadata storage
  • Request queues: Redis lists for FIFO queuing, hashes for request data and in-progress tracking, and Bloom filters for request deduplication

Hierarchy

Index

Methods

Methods

__init__

  • __init__(*, connection_string, redis, queue_dedup_strategy, queue_bloom_error_rate): None

  • Initialize the Redis storage client.

    Parameters

    • optionalkeyword-onlyconnection_string: str | None = None

      Redis connection string (e.g., "redis://localhost:6379"). Supports standard Redis URL format with optional database selection.

    • optionalkeyword-onlyredis: Redis | None = None

      Pre-configured Redis client instance.

    • optionalkeyword-onlyqueue_dedup_strategy: Literal[default, bloom] = 'default'

      Strategy for request queue deduplication. Options are:

      • 'default': Uses Redis sets for exact deduplication.
      • 'bloom': Uses Redis Bloom filters for probabilistic deduplication with lower memory usage. When using this approach, approximately 1 in 1e-7 requests will be falsely considered duplicate.
    • optionalkeyword-onlyqueue_bloom_error_rate: float = 1e-7

      Desired false positive rate for Bloom filter deduplication. Only relevant if queue_dedup_strategy is set to 'bloom'.

    Returns None

create_dataset_client

  • async create_dataset_client(*, id, name, alias, configuration): DatasetClient

create_kvs_client

create_rq_client

get_rate_limit_errors

  • get_rate_limit_errors(): dict[int, int]

get_storage_client_cache_key

  • get_storage_client_cache_key(configuration): Hashable

  • Return a cache key that can differentiate between different storages of this and other clients.

    Can be based on configuration or on the client itself. By default, returns a module and name of the client class.

    Parameters

    Returns Hashable

Page Options