Version: Next

Dataset <Data>

The Dataset class represents a store for structured data where each object stored has the same attributes, such as online store products or real estate offers. You can imagine it as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records. Typically it is used to store crawling results.

Do not instantiate this class directly, use the Dataset.open function instead.

Dataset stores its data either on local disk or in the Apify cloud, depending on whether the APIFY_LOCAL_STORAGE_DIR or APIFY_TOKEN environment variables are set.

If the APIFY_LOCAL_STORAGE_DIR environment variable is set, the data is stored in the local directory in the following files:

{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json

Note that {DATASET_ID} is the name or ID of the dataset. The default dataset has ID: default, unless you override it by setting the APIFY_DEFAULT_DATASET_ID environment variable. Each dataset item is stored as a separate JSON file, where {INDEX} is a zero-based index of the item in the dataset.

If the APIFY_TOKEN environment variable is set but APIFY_LOCAL_STORAGE_DIR not, the data is stored in the Apify Dataset cloud storage. Note that you can force usage of the cloud storage also by passing the forceCloud option to Dataset.open function, even if the APIFY_LOCAL_STORAGE_DIR variable is set.

Example usage:

// Write a single row to the default dataset
await Dataset.pushData({ col1: 123, col2: 'val2' });

// Open a named dataset
const dataset = await Dataset.open('some-name');

// Write a single row
await dataset.pushData({ foo: 'bar' });

// Write multiple rows
await dataset.pushData([
  { foo: 'bar2', col2: 'val2' },
  { col3: 123 },
]);

// Export the entirety of the dataset to one file in the key-value store
await dataset.exportToCSV('MY-DATA');

Index

Methods

drop
export
exportTo
exportToCSV
exportToJSON
forEach
getData
getInfo
map
pushData
reduce
exportToCSV
exportToJSON
getData
open

Properties

client

client: DatasetClient<Data>

readonlyconfig

config: Configuration = ...

id

id: string

log

log: Log = ...

optionalname

name?: string

Methods

drop

drop(): Promise<void>

Removes the dataset either from the Apify cloud storage or from the local directory, depending on the mode of operation.
Returns Promise<void>

export

export(options?: DatasetExportOptions): Promise<Data[]>

Returns all the data from the dataset. This will iterate through the whole dataset via the listItems() client method, which gives you only paginated results.
Parameters
- options: DatasetExportOptions = {}
Returns Promise<Data[]>

exportTo

exportTo(key: string, options?: DatasetExportToOptions, contentType?: string): Promise<Data[]>

Save the entirety of the dataset's contents into one file within a key-value store.
Parameters
- key: string
  The name of the value to save the data in.
- optionaloptions: DatasetExportToOptions
  An optional options object where you can provide the dataset and target KVS name.
- optionalcontentType: string
  Only JSON and CSV are supported currently, defaults to JSON.
Returns Promise<Data[]>

exportToCSV

exportToCSV(key: string, options?: Omit<DatasetExportToOptions, fromDataset>): Promise<void>

Save entire default dataset's contents into one CSV file within a key-value store.
Parameters
- key: string
  The name of the value to save the data in.
- optionaloptions: Omit<DatasetExportToOptions, fromDataset>
  An optional options object where you can provide the target KVS name.
Returns Promise<void>

exportToJSON

exportToJSON(key: string, options?: Omit<DatasetExportToOptions, fromDataset>): Promise<void>

Save entire default dataset's contents into one JSON file within a key-value store.
Parameters
- key: string
  The name of the value to save the data in.
- optionaloptions: Omit<DatasetExportToOptions, fromDataset>
  An optional options object where you can provide the target KVS name.
Returns Promise<void>

forEach

forEach(iteratee: DatasetConsumer<Data>, options?: DatasetIteratorOptions, index?: number): Promise<void>

Iterates over dataset items, yielding each in turn to an iteratee function. Each invocation of iteratee is called with two arguments: (item, index).

If the iteratee function returns a Promise then it is awaited before the next call. If it throws an error, the iteration is aborted and the forEach function throws the error.

Example usage
const dataset = await Dataset.open('my-results'); await dataset.forEach(async (item, index) => { console.log(`Item at ${index}: ${JSON.stringify(item)}`); });
Parameters
- iteratee: DatasetConsumer<Data>
  A function that is called for every item in the dataset.
- optionaloptions: DatasetIteratorOptions = {}
  All forEach() parameters.
- optionalindex: number = 0
  Specifies the initial index number passed to the iteratee function.
Returns Promise<void>

getData

getData(options?: DatasetDataOptions): Promise<DatasetContent<Data>>

Returns DatasetContent object holding the items in the dataset based on the provided parameters.
Parameters
- options: DatasetDataOptions = {}
Returns Promise<DatasetContent<Data>>

getInfo

getInfo(): Promise<undefined | DatasetInfo>

Returns an object containing general information about the dataset.

The function returns the same object as the Apify API Client's getDataset function, which in turn calls the Get dataset API endpoint.

Example:

{
  id: "WkzbQMuFYuamGv3YF",
  name: "my-dataset",
  userId: "wRsJZtadYvn4mBZmm",
  createdAt: new Date("2015-12-12T07:34:14.202Z"),
  modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
  accessedAt: new Date("2015-12-14T08:36:13.202Z"),
  itemCount: 14,
}

Returns Promise<undefined | DatasetInfo>

map

map(iteratee: DatasetMapper<Data, R>, options?: DatasetIteratorOptions): Promise<R[]>

Produces a new array of values by mapping each value in list through a transformation function iteratee(). Each invocation of iteratee() is called with two arguments: (element, index).

If iteratee returns a Promise then it's awaited before a next call.
Parameters
- iteratee: DatasetMapper<Data, R>
- optionaloptions: DatasetIteratorOptions = {}
  All map() parameters.
Returns Promise<R[]>

pushData

pushData(data: Data | Data[]): Promise<void>

Stores an object or an array of objects to the dataset. The function returns a promise that resolves when the operation finishes. It has no result, but throws on invalid args or other errors.

IMPORTANT: Make sure to use the await keyword when calling pushData(), otherwise the crawler process might finish before the data is stored!

The size of the data is limited by the receiving API and therefore pushData() will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, but the array itself may be of any size.

The function internally chunks the array into separate items and pushes them sequentially. The chunking process is stable (keeps order of data), but it does not provide a transaction safety mechanism. Therefore, in the event of an uploading error (after several automatic retries), the function's Promise will reject and the dataset will be left in a state where some of the items have already been saved to the dataset while other items from the source array were not. To overcome this limitation, the developer may, for example, read the last item saved in the dataset and re-attempt the save of the data from this item onwards to prevent duplicates.
Parameters
- data: Data | Data[]
  Object or array of objects containing data to be stored in the default dataset. The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
Returns Promise<void>

reduce

reduce(iteratee: DatasetReducer<Data, Data>): Promise<undefined | Data>
reduce(iteratee: DatasetReducer<Data, Data>, memo: undefined, options?: DatasetIteratorOptions): Promise<undefined | Data>
reduce(iteratee: DatasetReducer<T, Data>, memo: T, options?: DatasetIteratorOptions): Promise<T>

Reduces a list of values down to a single value.

The first element of the dataset is the initial value, with each successive reductions should be returned by iteratee(). The iteratee() is passed three arguments: the memo, value and index of the current element being folded into the reduction.

The iteratee is first invoked on the second element of the list (index = 1), with the first element given as the memo parameter. After that, the rest of the elements in the dataset is passed to iteratee, with the result of the previous invocation as the memo.

If iteratee() returns a Promise it's awaited before a next call.

If the dataset is empty, reduce will return undefined.
Parameters
- iteratee: DatasetReducer<Data, Data>
Returns Promise<undefined | Data>

staticexportToCSV

exportToCSV(key: string, options?: DatasetExportToOptions): Promise<void>

Save entire default dataset's contents into one CSV file within a key-value store.
Parameters
- key: string
  The name of the value to save the data in.
- optionaloptions: DatasetExportToOptions
  An optional options object where you can provide the dataset and target KVS name.
Returns Promise<void>

staticexportToJSON

exportToJSON(key: string, options?: DatasetExportToOptions): Promise<void>

Save entire default dataset's contents into one JSON file within a key-value store.
Parameters
- key: string
  The name of the value to save the data in.
- optionaloptions: DatasetExportToOptions
  An optional options object where you can provide the dataset and target KVS name.
Returns Promise<void>

staticgetData

getData(options?: DatasetDataOptions): Promise<DatasetContent<Data>>

Returns DatasetContent object holding the items in the dataset based on the provided parameters.
Parameters
- options: DatasetDataOptions = {}
Returns Promise<DatasetContent<Data>>

staticopen

open(datasetIdOrName?: null | string, options?: StorageManagerOptions): Promise<Dataset<Data>>

Opens a dataset and returns a promise resolving to an instance of the Dataset class.

Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. The actual data is stored either on the local filesystem or in the cloud.

For more details and code examples, see the Dataset class.
Parameters
- optionaldatasetIdOrName: null | string
  ID or name of the dataset to be opened. If null or undefined, the function returns the default dataset associated with the crawler run.
- optionaloptions: StorageManagerOptions = {}
  Storage manager options.
Returns Promise<Dataset<Data>>

Index

Properties

Methods

Properties

client

readonlyconfig

id

log

optionalname

Methods

drop

Returns Promise<void>

export

Parameters

options: DatasetExportOptions = {}

Returns Promise<Data[]>

exportTo

Parameters

key: string

optionaloptions: DatasetExportToOptions

optionalcontentType: string

Returns Promise<Data[]>

exportToCSV

Parameters

key: string

optionaloptions: Omit<DatasetExportToOptions, fromDataset>

Returns Promise<void>

exportToJSON

Parameters

key: string

optionaloptions: Omit<DatasetExportToOptions, fromDataset>

Returns Promise<void>

forEach

Parameters

iteratee: DatasetConsumer<Data>

optionaloptions: DatasetIteratorOptions = {}

optionalindex: number = 0

Returns Promise<void>

getData

Parameters

options: DatasetDataOptions = {}

Returns Promise<DatasetContent<Data>>

getInfo

Returns Promise<undefined | DatasetInfo>

map

Parameters

iteratee: DatasetMapper<Data, R>

optionaloptions: DatasetIteratorOptions = {}

Returns Promise<R[]>

pushData

Parameters

data: Data | Data[]

Returns Promise<void>

reduce

Parameters

iteratee: DatasetReducer<Data, Data>

Returns Promise<undefined | Data>

staticexportToCSV

Parameters

key: string

optionaloptions: DatasetExportToOptions

Returns Promise<void>

staticexportToJSON

Parameters

key: string

optionaloptions: DatasetExportToOptions

Returns Promise<void>

staticgetData

Parameters

options: DatasetDataOptions = {}

Returns Promise<DatasetContent<Data>>

staticopen

Parameters

optionaldatasetIdOrName: null | string

optionaloptions: StorageManagerOptions = {}

Returns Promise<Dataset<Data>>