Skip to main content
Version: Next

MemoryStorageBackend

Represents a storage backend capable of working with datasets, key-value stores and request queues.

A new storage backend needs to implement 4 classes:

  • StorageBackend - the factory that creates sub-backends
  • DatasetBackend - operations on a single dataset
  • KeyValueStoreBackend - operations on a single key-value store
  • RequestQueueBackend - operations on a single request queue

The StorageBackend acts as an async factory: each create* method either opens an existing storage or creates a new one, returning a sub-backend bound to that storage instance.

Implements

Index

Constructors

constructor

Properties

optionalreadonlylogger

logger?: CrawleeLogger

Methods

createDatasetBackend

  • Create (or open) a dataset backend. If id is provided, opens the dataset with that ID. If name is provided, opens an existing dataset with that name or creates a new one. If neither is provided, opens or creates the default dataset.


    Parameters

    Returns Promise<DatasetBackend<Dictionary>>

createKeyValueStoreBackend

  • Create (or open) a key-value store backend. If id is provided, opens the key-value store with that ID. If name is provided, opens an existing store with that name or creates a new one. If neither is provided, opens or creates the default key-value store.


    Parameters

    Returns Promise<KeyValueStoreBackend>

createRequestQueueBackend

  • createRequestQueueBackend(options): Promise<RequestQueueBackend>
  • Create (or open) a request queue backend. If id is provided, opens the request queue with that ID. If name is provided, opens an existing queue with that name or creates a new one. If neither is provided, opens or creates the default request queue.


    Parameters

    Returns Promise<RequestQueueBackend>

getStorageBackendCacheKey

  • getStorageBackendCacheKey(): string
  • Return a per-instance unique cache key so that distinct MemoryStorageBackend instances get separate cache partitions in the storage backend cache.


    Returns string

purge

  • purge(): Promise<void>
  • Cleans up the run-scoped storages before the run starts. For the in-memory storage this simply resets the in-memory state of the cached backends.


    Returns Promise<void>

storageExists

  • storageExists(id, type): Promise<boolean>
  • Check whether a storage with the given ID exists.

    Used internally to resolve ambiguous idOrName strings passed to Dataset.open(), KeyValueStore.open(), and RequestQueue.open().


    Parameters

    • id: string
    • type: Dataset | KeyValueStore | RequestQueue

    Returns Promise<boolean>

teardown

  • teardown(): Promise<void>
  • This method should be called at the end of the process. The in-memory storage holds no resources that outlive the process (no file handles, no cross-process locks), so there is nothing to do.


    Returns Promise<void>