StorageBackend

Represents a storage backend capable of working with datasets, key-value stores and request queues.

A new storage backend needs to implement 4 classes:

• StorageBackend - the factory that creates sub-backends
• DatasetBackend - operations on a single dataset
• KeyValueStoreBackend - operations on a single key-value store
• RequestQueueBackend - operations on a single request queue

The StorageBackend acts as an async factory: each create* method either opens an existing storage or creates a new one, returning a sub-backend bound to that storage instance.

Implemented by

• FileSystemStorageBackend
• MemoryStorageBackend

Index

Properties

• stats

Methods

• createDatasetBackend
• createKeyValueStoreBackend
• createRequestQueueBackend
• getStorageBackendCacheKey
• purge
• storageExists
• teardown

Properties

optionalstats

stats?: { rateLimitErrors: number[] }

Type declaration

• rateLimitErrors: number[]

Methods

createDatasetBackend

• createDatasetBackend(options): Promise<DatasetBackend<Dictionary>>

• Create (or open) a dataset backend. If id is provided, opens the dataset with that ID. If name is provided, opens an existing dataset with that name or creates a new one. If neither is provided, opens or creates the default dataset.

  ---

  Parameters

  • optionaloptions: StorageIdentifier

  Returns Promise<DatasetBackend<Dictionary>>

createKeyValueStoreBackend

• createKeyValueStoreBackend(options): Promise<KeyValueStoreBackend>

• Create (or open) a key-value store backend. If id is provided, opens the key-value store with that ID. If name is provided, opens an existing store with that name or creates a new one. If neither is provided, opens or creates the default key-value store.

  ---

  Parameters

  • optionaloptions: StorageIdentifier

  Returns Promise<KeyValueStoreBackend>

createRequestQueueBackend

• createRequestQueueBackend(options): Promise<RequestQueueBackend>

• Create (or open) a request queue backend. If id is provided, opens the request queue with that ID. If name is provided, opens an existing queue with that name or creates a new one. If neither is provided, opens or creates the default request queue.

  ---

  Parameters

  • optionaloptions: StorageIdentifier

  Returns Promise<RequestQueueBackend>

optionalgetStorageBackendCacheKey

• getStorageBackendCacheKey(): string

• Return an opaque key that uniquely identifies this storage backend instance.

  The key is used by StorageInstanceManager to partition the storage cache per-backend, so that two storages with the same name but backed by different clients (e.g. a local MemoryStorageBackend and a cloud ApifyClient) are cached as separate instances.

  When not provided, the fallback uses the client's constructor name, so different StorageBackend implementations automatically get separate cache partitions.

  ---

  Returns string

optionalpurge

• purge(): Promise<void>

• Empty the run-scoped storages — the default one and every alias-keyed one, including any left behind by a previous run. Named storages persist across runs, as does the default store's INPUT.

  ---

  Returns Promise<void>

optionalstorageExists

• storageExists(id, type): Promise<boolean>

• Check whether a storage with the given ID exists.

  Used internally to resolve ambiguous idOrName strings passed to Dataset.open(), KeyValueStore.open(), and RequestQueue.open().

  ---

  Parameters

  • id: string

  • type: Dataset | KeyValueStore | RequestQueue

  Returns Promise<boolean>

optionalteardown

• teardown(): Promise<void>

• Returns Promise<void>