FileSystemStorageBackend

A file-system storage backend backed by the native @crawlee/fs-storage-native Rust extension.

The native extension owns the on-disk format, timestamps, item counting, request-queue locking and state persistence. This class is responsible for resolving the user-facing id / name / alias identifiers to native storages, caching the opened backends (so that storageExists, purge and teardown can operate over them), and exposing them through the @crawlee/types interfaces.

Implements

• StorageBackend

Index

Constructors

• constructor

Properties

• datasetsDirectory
• keyValueStoresDirectory
• localDataDirectory
• logger
• requestQueueAccess
• requestQueuesDirectory

Methods

• createDatasetBackend
• createKeyValueStoreBackend
• createRequestQueueBackend
• getStorageBackendCacheKey
• purge
• storageExists
• teardown

Constructors

constructor

• new FileSystemStorageBackend(options): FileSystemStorageBackend

• Parameters

  • options: FileSystemStorageOptions

  Returns FileSystemStorageBackend

Properties

readonlydatasetsDirectory

datasetsDirectory: string

readonlykeyValueStoresDirectory

keyValueStoresDirectory: string

readonlylocalDataDirectory

localDataDirectory: string

optionalreadonlylogger

logger?: CrawleeLogger

readonlyrequestQueueAccess

requestQueueAccess: single | shared

readonlyrequestQueuesDirectory

requestQueuesDirectory: string

Methods

createDatasetBackend

• createDatasetBackend(options): Promise<DatasetBackend<Dictionary>>

• Implementation of storage.StorageBackend.createDatasetBackend

  Create (or open) a dataset backend. If id is provided, opens the dataset with that ID. If name is provided, opens an existing dataset with that name or creates a new one. If neither is provided, opens or creates the default dataset.

  ---

  Parameters

  • options: StorageIdentifier = {}

  Returns Promise<DatasetBackend<Dictionary>>

createKeyValueStoreBackend

• createKeyValueStoreBackend(options): Promise<KeyValueStoreBackend>

• Implementation of storage.StorageBackend.createKeyValueStoreBackend

  Create (or open) a key-value store backend. If id is provided, opens the key-value store with that ID. If name is provided, opens an existing store with that name or creates a new one. If neither is provided, opens or creates the default key-value store.

  ---

  Parameters

  • options: StorageIdentifier = {}

  Returns Promise<KeyValueStoreBackend>

createRequestQueueBackend

• createRequestQueueBackend(options): Promise<RequestQueueBackend>

• Implementation of storage.StorageBackend.createRequestQueueBackend

  Create (or open) a request queue backend. If id is provided, opens the request queue with that ID. If name is provided, opens an existing queue with that name or creates a new one. If neither is provided, opens or creates the default request queue.

  ---

  Parameters

  • options: StorageIdentifier = {}

  Returns Promise<RequestQueueBackend>

getStorageBackendCacheKey

• getStorageBackendCacheKey(): string

• Implementation of storage.StorageBackend.getStorageBackendCacheKey

  Return a cache key that includes the resolved storage directory, so that two FileSystemStorageBackend instances pointing at different directories get separate cache partitions, by including the storage directory in the cache key.

  ---

  Returns string

purge

• purge(): Promise<void>

• Implementation of storage.StorageBackend.purge

  Cleans up the run-scoped storages before the run starts, sweeping the storage directories so that leftovers from a previous process are caught too.

  ---

  Returns Promise<void>

storageExists

• storageExists(id, type): Promise<boolean>

• Implementation of storage.StorageBackend.storageExists

  Check whether a storage with the given ID exists.

  Used internally to resolve ambiguous idOrName strings passed to Dataset.open(), KeyValueStore.open(), and RequestQueue.open().

  ---

  Parameters

  • id: string

  • type: Dataset | KeyValueStore | RequestQueue

  Returns Promise<boolean>

teardown

• teardown(): Promise<void>

• Implementation of storage.StorageBackend.teardown

  This method should be called at the end of the process, to ensure all data is saved.

  It persists the state of every opened request queue so that requests fetched but not yet handled are not stuck (until their lock expires) for the next consumer of the same on-disk queue.

  ---

  Returns Promise<void>