Skip to main content

BaseDatasetCollectionClient

crawlee.base_storage_client._base_dataset_collection_client.BaseDatasetCollectionClient

Abstract base class for dataset collection clients.

This collection client handles operations that involve multiple instances of a given resource type.

Index

Methods

get_or_create

  • async get_or_create(*, id, name, schema): DatasetMetadata
  • Retrieve an existing dataset by its name or ID, or create a new one if it does not exist.


    Parameters

    • id: str | None = Nonekeyword-only
    • name: str | None = Nonekeyword-only
    • schema: dict | None = Nonekeyword-only

    Returns DatasetMetadata

list

  • async list(*, unnamed, limit, offset, desc): DatasetListPage
  • List the available datasets.


    Parameters

    • unnamed: bool = Falsekeyword-only
    • limit: int | None = Nonekeyword-only
    • offset: int | None = Nonekeyword-only
    • desc: bool = Falsekeyword-only

    Returns DatasetListPage