Skip to main content
Version: Next

Dataset <Data>

The Dataset class represents a store for structured data where each object stored has the same attributes, such as online store products or real estate offers. You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records. Typically it is used to store crawling results.

Do not instantiate this class directly, use the Dataset.open function instead.

Dataset stores its data either on local disk or in the Apify cloud, depending on whether the APIFY_LOCAL_STORAGE_DIR or APIFY_TOKEN environment variables are set.

If the APIFY_LOCAL_STORAGE_DIR environment variable is set, the data is stored in the local directory in the following files:

{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json

Note that {DATASET_ID} is the name or ID of the dataset. The default dataset has ID: default, unless you override it by setting the APIFY_DEFAULT_DATASET_ID environment variable. Each dataset item is stored as a separate JSON file, where {INDEX} is a zero-based index of the item in the dataset.

If the APIFY_TOKEN environment variable is set but APIFY_LOCAL_STORAGE_DIR not, the data is stored in the Apify Dataset cloud storage. Note that you can force usage of the cloud storage also by passing the forceCloud option to Dataset.open function, even if the APIFY_LOCAL_STORAGE_DIR variable is set.

Example usage:

// Write a single row to the default dataset
await Dataset.pushData({ col1: 123, col2: 'val2' });

// Open a named dataset
const dataset = await Dataset.open('some-name');

// Write a single row
await dataset.pushData({ foo: 'bar' });

// Write multiple rows
await dataset.pushData([
  { foo: 'bar2', col2: 'val2' },
  { col3: 123 },
]);

// Export the entirety of the dataset to one file in the key-value store
await dataset.exportToCSV('MY-DATA');

Index

Properties

Accessors

Methods

Properties

backend

backend: DatasetBackend<Data>

readonlyconfiguration

configuration: Configuration = ...

id

id: string

log

log: CrawleeLogger

optionalname

name?: string

Accessors

stats

  • Backend-independent usage counters tracked for this dataset (read / write operations issued to the underlying storage backend). Counted per backend call.

    Returns DatasetStats

Methods

[asyncIterator]

  • [asyncIterator](): AsyncGenerator<Data, void, undefined>

  • Default async iterator for the dataset, iterating over items. Allows using the dataset directly in a for await...of loop.

    Example usage:

    const dataset = await Dataset.open('my-results');
    for await (const item of dataset) {
      console.log(item);
    }

    Returns AsyncGenerator<Data, void, undefined>

drop

  • drop(): Promise<void>

  • Removes the dataset either from the Apify cloud storage or from the local directory, depending on the mode of operation.

    Returns Promise<void>

entries

  • entries(options): AsyncIterable<[number, Data], any, any> & Promise<[number, Data][]>

  • Returns dataset entries (index-value pairs).

    When awaited (await dataset.entries()), returns all entries as a flat [index, item][] array. When used as an async iterable (for await...of), iterates over all entries across pages without loading everything into memory at once.

    Example usage:

    const dataset = await Dataset.open('my-results');

    // Iterate over all entries
    for await (const [index, item] of dataset.entries()) {
      console.log(`Item at ${index}: ${JSON.stringify(item)}`);
    }

    // Or fetch all at once
    const entries = await dataset.entries();
    console.log(entries);

    Parameters

    Returns AsyncIterable<[number, Data], any, any> & Promise<[number, Data][]>

export

  • export(options): Promise<Data[]>

  • Returns all the data from the dataset. This will iterate through the whole dataset via the listItems() client method, which gives you only paginated results.

    Parameters

    Returns Promise<Data[]>

exportTo

  • exportTo(key, options, contentType): Promise<Data[]>

  • Save the entirety of the dataset's contents into one file within a key-value store.

    Parameters

    • key: string

      The name of the value to save the data in.

    • optionaloptions: DatasetExportToOptions

      An optional options object where you can provide the dataset and target KVS name.

    • optionalcontentType: string

      Only JSON and CSV are supported currently, defaults to JSON.

    Returns Promise<Data[]>

exportToCSV

  • exportToCSV(key, options): Promise<void>

  • Save entire default dataset's contents into one CSV file within a key-value store.

    Parameters

    • key: string

      The name of the value to save the data in.

    • optionaloptions: Omit<DatasetExportToOptions, fromDataset>

      An optional options object where you can provide the target KVS name.

    Returns Promise<void>

exportToJSON

  • exportToJSON(key, options): Promise<void>

  • Save entire default dataset's contents into one JSON file within a key-value store.

    Parameters

    • key: string

      The name of the value to save the data in.

    • optionaloptions: Omit<DatasetExportToOptions, fromDataset>

      An optional options object where you can provide the target KVS name.

    Returns Promise<void>

forEach

  • forEach(iteratee, options, index): Promise<void>

  • Iterates over dataset items, yielding each in turn to an iteratee function. Each invocation of iteratee is called with two arguments: (item, index).

    If the iteratee function returns a Promise then it is awaited before the next call. If it throws an error, the iteration is aborted and the forEach function throws the error.

    Example usage

    const dataset = await Dataset.open('my-results');
    await dataset.forEach(async (item, index) => {
      console.log(`Item at ${index}: ${JSON.stringify(item)}`);
    });

    Parameters

    • iteratee: DatasetConsumer<Data>

      A function that is called for every item in the dataset.

    • optionaloptions: DatasetIteratorOptions = {}

      All forEach() parameters.

    • optionalindex: number = 0

      Specifies the initial index number passed to the iteratee function.

    Returns Promise<void>

getData

getInfo

  • Returns an object containing general information about the dataset.

    Example:

    {
      id: "WkzbQMuFYuamGv3YF",
      name: "my-dataset",
      createdAt: new Date("2015-12-12T07:34:14.202Z"),
      modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
      accessedAt: new Date("2015-12-14T08:36:13.202Z"),
      itemCount: 14,
    }

    Returns Promise<DatasetInfo>

    Throws - If the underlying storage no longer exists (e.g. it was deleted externally).

map

  • map<R>(iteratee, options): Promise<R[]>

  • Produces a new array of values by mapping each value in list through a transformation function iteratee(). Each invocation of iteratee() is called with two arguments: (element, index).

    If iteratee returns a Promise then it's awaited before a next call.

    Parameters

    Returns Promise<R[]>

purge

  • purge(): Promise<void>

  • Removes all items from the dataset but keeps the dataset itself, along with its id and name.

    Returns Promise<void>

pushData

  • pushData(data): Promise<void>

  • Stores an object or an array of objects to the dataset. The function returns a promise that resolves when the operation finishes. It has no result, but throws on invalid args or other errors.

    IMPORTANT: Make sure to use the await keyword when calling pushData(), otherwise the crawler process might finish before the data is stored!

    Parameters

    • data: Data | Data[]

      Object or array of objects containing data to be stored in the default dataset. The objects must be serializable to JSON.

    Returns Promise<void>

reduce

  • reduce(iteratee): Promise<undefined | Data>
  • reduce(iteratee, memo, options): Promise<undefined | Data>
  • reduce<T>(iteratee, memo, options): Promise<T>

  • Reduces a list of values down to a single value.

    The first element of the dataset is the initial value, with each successive reductions should be returned by iteratee(). The iteratee() is passed three arguments: the memo, value and index of the current element being folded into the reduction.

    The iteratee is first invoked on the second element of the list (index = 1), with the first element given as the memo parameter. After that, the rest of the elements in the dataset is passed to iteratee, with the result of the previous invocation as the memo.

    If iteratee() returns a Promise it's awaited before a next call.

    If the dataset is empty, reduce will return undefined.

    Parameters

    Returns Promise<undefined | Data>

values

  • values(options): AsyncIterable<Data, any, any> & Promise<Data[]>

  • Returns dataset items.

    When awaited (await dataset.values()), returns all items as a flat Data[] array. When used as an async iterable (for await...of), iterates over all items across pages without loading everything into memory at once.

    Example usage:

    const dataset = await Dataset.open('my-results');

    // Iterate over all items (memory-efficient for large datasets)
    for await (const item of dataset.values()) {
      console.log(item);
    }

    // Or fetch all items at once
    const items = await dataset.values();
    console.log(items);

    Parameters

    Returns AsyncIterable<Data, any, any> & Promise<Data[]>

staticexportToCSV

  • exportToCSV(key, options): Promise<void>

  • Save entire default dataset's contents into one CSV file within a key-value store.

    Parameters

    • key: string

      The name of the value to save the data in.

    • optionaloptions: DatasetExportToOptions

      An optional options object where you can provide the dataset and target KVS name.

    Returns Promise<void>

staticexportToJSON

  • exportToJSON(key, options): Promise<void>

  • Save entire default dataset's contents into one JSON file within a key-value store.

    Parameters

    • key: string

      The name of the value to save the data in.

    • optionaloptions: DatasetExportToOptions

      An optional options object where you can provide the dataset and target KVS name.

    Returns Promise<void>

staticgetData

staticopen

  • open<Data>(identifier, options): Promise<Dataset<Data>>

  • Opens a dataset and returns a promise resolving to an instance of the Dataset class.

    Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The actual data is stored either on the local filesystem or in the cloud.

    For more details and code examples, see the Dataset class.

    Parameters

    • optionalidentifier: null | string | StorageIdentifier

      ID or name of the dataset to be opened. If a string is provided, it will first be looked up as an ID; if no such storage exists, it will be treated as a name. If null or undefined, the function returns the default dataset associated with the crawler run.

    • optionaloptions: StorageOpenOptions = {}

      Storage manager options.

    Returns Promise<Dataset<Data>>

Page Options