crawlee

Index

Classes

• AutoscaledPool
• BasicCrawler
• BeautifulSoupCrawler
• BrowserPool
• ContextPipeline
• CurlImpersonateHttpClient
• Dataset
• EventManager
• HeaderGenerator
• HttpCrawler
• HttpxHttpClient
• KeyValueStore
• LocalEventManager
• MemoryStorageClient
• ParselCrawler
• PlaywrightBrowserController
• PlaywrightBrowserPlugin
• PlaywrightCrawler
• ProxyConfiguration
• RequestList
• RequestQueue
• Router
• SessionPool
• Snapshotter
• Statistics
• SystemStatus

Abstract classes

• BaseDatasetClient
• BaseDatasetCollectionClient
• BaseHttpClient
• BaseKeyValueStoreClient
• BaseKeyValueStoreCollectionClient
• BaseRequestQueueClient
• BaseRequestQueueCollectionClient
• BaseStorageClient
• RequestProvider

Data structures

• BasicCrawlerOptions
• BasicCrawlingContext
• BatchRequestsOperationResponse
• BeautifulSoupCrawlingContext
• ConcurrencySettings
• Configuration
• DatasetItemsListPage
• DatasetListPage
• DatasetMetadata
• EnqueueStrategy
• FinalStatistics
• HttpCrawlingContext
• HttpCrawlingResult
• HttpResponse
• KeyValueStoreKeyInfo
• KeyValueStoreListKeysPage
• KeyValueStoreListPage
• KeyValueStoreMetadata
• KeyValueStoreRecord
• KeyValueStoreRecordMetadata
• ParselCrawlingContext
• PlaywrightCrawlingContext
• PlaywrightPreNavigationContext
• ProcessedRequest
• ProlongRequestLockResponse
• ProxyInfo
• Request
• RequestQueueHead
• RequestQueueHeadState
• RequestQueueHeadWithLocks
• RequestQueueListPage
• RequestQueueMetadata
• Session
• StatisticsPersistedState
• StatisticsState
• UnprocessedRequest

Errors

• ContextPipelineFinalizationError
• ContextPipelineInitializationError
• ContextPipelineInterruptedError
• HttpStatusCodeError
• ProxyError
• RequestHandlerError
• ServiceConflictError
• SessionError
• UserDefinedErrorHandlerError

Functions

• get_configuration
• get_configuration_if_set
• get_event_manager
• get_storage_client
• set_cloud_storage_client
• set_configuration
• set_default_storage_client_type
• set_event_manager
• set_local_storage_client

Functions

get_configuration

• get_configuration(): Configuration

• Get the configuration object.

  ---

  Returns Configuration

get_configuration_if_set

• get_configuration_if_set(): Configuration | None

• Get the configuration object, or None if it hasn't been set yet.

  ---

  Returns Configuration | None

get_event_manager

• get_event_manager(): EventManager

• Get the event manager.

  ---

  Returns EventManager

get_storage_client

• get_storage_client(*, client_type): BaseStorageClient

• Get the storage client instance for the current environment.

  ---

  Parameters

  • optionalkeyword-onlyclient_type: StorageClientType | None = None

    Allows retrieving a specific storage client type, regardless of where we are running.

  Returns BaseStorageClient

set_cloud_storage_client

• set_cloud_storage_client(*, cloud_client): None

• Set the cloud storage client instance.

  ---

  Parameters

  • optionalkeyword-onlycloud_client: BaseStorageClient

    The cloud storage client instance.

  Returns None

set_configuration

• set_configuration(*, configuration): None

• Set the configuration object.

  ---

  Parameters

  • optionalkeyword-onlyconfiguration: Configuration

  Returns None

set_default_storage_client_type

• set_default_storage_client_type(*, client_type): None

• Set the default storage client type.

  ---

  Parameters

  • optionalkeyword-onlyclient_type: StorageClientType

  Returns None

set_event_manager

• set_event_manager(*, event_manager): None

• Set the event manager.

  ---

  Parameters

  • optionalkeyword-onlyevent_manager: EventManager

  Returns None

set_local_storage_client

• set_local_storage_client(*, local_client): None

• Set the local storage client instance.

  ---

  Parameters

  • optionalkeyword-onlylocal_client: BaseStorageClient

    The local storage client instance.

  Returns None