KeyValueStore

The KeyValueStore class represents a key-value store, a simple data storage that is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME content type. Key-value stores are ideal for saving screenshots, crawler inputs and outputs, web pages, PDFs or to persist the state of crawlers.

Do not instantiate this class directly, use the KeyValueStore.open function instead.

Each crawler run is associated with a default key-value store, which is created exclusively for the run. By convention, the crawler input and output are stored into the default key-value store under the INPUT and OUTPUT key, respectively. Typically, input and output are JSON files, although it can be any other format. To access the default key-value store directly, you can use the KeyValueStore.getValue and KeyValueStore.setValue convenience functions.

To access the input, you can also use the KeyValueStore.getInput convenience function.

KeyValueStore stores its data on a local disk.

If the CRAWLEE_STORAGE_DIR environment variable is set, the data is stored in the local directory in the following files:

{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{INDEX}.{EXT}

Note that {STORE_ID} is the name or ID of the key-value store. The default key-value store has ID: default, unless you override it by setting the CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID environment variable. The {KEY} is the key of the record and {EXT} corresponds to the MIME content type of the data value.

Example usage:

// Get crawler input from the default key-value store.
const input = await KeyValueStore.getInput();
// Get some value from the default key-value store.
const otherValue = await KeyValueStore.getValue('my-key');

// Write crawler output to the default key-value store.
await KeyValueStore.setValue('OUTPUT', { myResult: 123 });

// Open a named key-value store
const store = await KeyValueStore.open('some-name');

// Write a record. JavaScript object is automatically converted to JSON,
// strings and binary buffers are stored as they are
await store.setValue('some-key', { foo: 'bar' });

// Read a record. Note that JSON is automatically parsed to a JavaScript object,
// text data returned as a string and other data is returned as binary buffer
const value = await store.getValue('some-key');

 // Drop (delete) the store
await store.drop();

Index

Properties

• config
• id
• name

Methods

• drop
• forEachKey
• getAutoSavedValue
• getPublicUrl
• getValue
• recordExists
• setValue
• getAutoSavedValue
• open
• recordExists

Properties

readonlyconfig

config: Configuration = ...

readonlyid

id: string

optionalreadonlyname

name?: string

Methods

drop

• drop(): Promise<void>

• Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.

  ---

  Returns Promise<void>

forEachKey

• forEachKey(iteratee: KeyConsumer, options?: KeyValueStoreIteratorOptions): Promise<void>

• Iterates over key-value store keys, yielding each in turn to an iteratee function. Each invocation of iteratee is called with three arguments: (key, index, info), where key is the record key, index is a zero-based index of the key in the current iteration (regardless of options.exclusiveStartKey) and info is an object that contains a single property size indicating size of the record in bytes.

  If the iteratee function returns a Promise then it is awaited before the next call. If it throws an error, the iteration is aborted and the forEachKey function throws the error.

  Example usage

  const keyValueStore = await KeyValueStore.open();
  await keyValueStore.forEachKey(async (key, index, info) => {
    console.log(`Key at ${index}: ${key} has size ${info.size}`);
  });

  ---

  Parameters

  • iteratee: KeyConsumer

    A function that is called for every key in the key-value store.

  • optionaloptions: KeyValueStoreIteratorOptions = {}

    All forEachKey() parameters.

  Returns Promise<void>

getAutoSavedValue

• getAutoSavedValue<T>(key: string, defaultValue?: T): Promise<T>

• Type parameters

  • T: Dictionary = Dictionary

  Parameters

  • key: string

  • defaultValue: T = ...

  Returns Promise<T>

getPublicUrl

• getPublicUrl(key: string): string

• Returns a file URL for the given key.

  ---

  Parameters

  • key: string

  Returns string

getValue

• getValue<T>(key: string): Promise<null | T>
• getValue<T>(key: string, defaultValue: T): Promise<T>

• Gets a value from the key-value store.

  The function returns a Promise that resolves to the record value, whose JavaScript type depends on the MIME content type of the record. Records with the application/json content type are automatically parsed and returned as a JavaScript object. Similarly, records with text/plain content types are returned as a string. For all other content types, the value is returned as a raw Buffer instance.

  If the record does not exist, the function resolves to null.

  To save or delete a value in the key-value store, use the KeyValueStore.setValue function.

  Example usage:

  const store = await KeyValueStore.open();
  const buffer = await store.getValue('screenshot1.png');

  ---

  Type parameters

  • T = unknown

  Parameters

  • key: string

    Unique key of the record. It can be at most 256 characters long and only consist of the following characters: a-z, A-Z, 0-9 and !-_.'()

  Returns Promise<null | T>

  Returns a promise that resolves to an object, string or Buffer, depending on the MIME content type of the record.

recordExists

• recordExists(key: string): Promise<boolean>

• Tests whether a record with the given key exists in the key-value store without retrieving its value.

  ---

  Parameters

  • key: string

    The queried record key.

  Returns Promise<boolean>

  true if the record exists, false if it does not.

setValue

• setValue<T>(key: string, value: null | T, options?: RecordOptions): Promise<void>

• Saves or deletes a record in the key-value store. The function returns a promise that resolves once the record has been saved or deleted.

  Example usage:

  const store = await KeyValueStore.open();
  await store.setValue('OUTPUT', { foo: 'bar' });

  Beware that the key can be at most 256 characters long and only contain the following characters: a-zA-Z0-9!-_.'()

  By default, value is converted to JSON and stored with the application/json; charset=utf-8 MIME content type. To store the value with another content type, pass it in the options as follows:

  const store = await KeyValueStore.open('my-text-store');
  await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });

  If you set custom content type, value must be either a string or Buffer, otherwise an error will be thrown.

  If value is null, the record is deleted instead. Note that the setValue() function succeeds regardless whether the record existed or not.

  To retrieve a value from the key-value store, use the KeyValueStore.getValue function.

  IMPORTANT: Always make sure to use the await keyword when calling setValue(), otherwise the crawler process might finish before the value is stored!

  ---

  Type parameters

  • T

  Parameters

  • key: string

    Unique key of the record. It can be at most 256 characters long and only consist of the following characters: a-z, A-Z, 0-9 and !-_.'()

  • value: null | T

    Record data, which can be one of the following values:

    • If null, the record in the key-value store is deleted.
    • If no options.contentType is specified, value can be any JavaScript object and it will be stringified to JSON.
    • If options.contentType is set, value is taken as is and it must be a String or Buffer. For any other value an error will be thrown.

  • optionaloptions: RecordOptions = {}

    Record options.

  Returns Promise<void>

staticgetAutoSavedValue

• getAutoSavedValue<T>(key: string, defaultValue?: T): Promise<T>

• Type parameters

  • T: Dictionary = Dictionary

  Parameters

  • key: string

  • defaultValue: T = ...

  Returns Promise<T>

staticopen

• open(storeIdOrName?: null | string, options?: StorageManagerOptions): Promise<KeyValueStore>

• Opens a key-value store and returns a promise resolving to an instance of the KeyValueStore class.

  Key-value stores are used to store records or files, along with their MIME content type. The records are stored and retrieved using a unique key. The actual data is stored either on a local filesystem or in the Apify cloud.

  For more details and code examples, see the KeyValueStore class.

  ---

  Parameters

  • optionalstoreIdOrName: null | string

    ID or name of the key-value store to be opened. If null or undefined, the function returns the default key-value store associated with the crawler run.

  • optionaloptions: StorageManagerOptions = {}

    Storage manager options.

  Returns Promise<KeyValueStore>

staticrecordExists

• recordExists(key: string): Promise<boolean>

• Tests whether a record with the given key exists in the default KeyValueStore associated with the current crawler run.

  ---

  Parameters

  • key: string

    The queried record key.

  Returns Promise<boolean>

  true if the record exists, false if it does not.