Skip to main content

Upgrading to v1

This page summarizes the breaking changes between Crawlee for Python v0.6 and v1.0.

Terminology change: "browser" in different contexts​

The word "browser" is now used distinctly in two contexts:

  • Playwright context - Refers to Playwright-supported browsers (chromium, firefox, webkit, edge).
  • Fingerprinting context - Refers to browsers supported by fingerprint generation (chrome, firefox, safari, edge).

The type of HeaderGeneratorOptions.browsers has changed accordingly:

Before (v0.6):

from crawlee.fingerprint_suite import HeaderGeneratorOptions

HeaderGeneratorOptions(browsers=['chromium'])
HeaderGeneratorOptions(browsers=['webkit'])

Now (v1.0):

from crawlee.fingerprint_suite import HeaderGeneratorOptions

HeaderGeneratorOptions(browsers=['chrome'])
HeaderGeneratorOptions(browsers=['safari'])

New default HTTP client​

Crawlee v1.0 now uses ImpitHttpClient (based on impit library) as the default HTTP client, replacing HttpxHttpClient (based on httpx library).

If you want to keep using HttpxHttpClient, install Crawlee with httpx extra, e.g. using pip:

pip install 'crawlee[httpx]'

And then provide the HTTP client explicitly to the crawler:

from crawlee.crawlers import HttpCrawler
from crawlee.http_clients import HttpxHttpClient

client = HttpxHttpClient()
crawler = HttpCrawler(http_client=client)

See the HTTP clients guide for all options.

Changes in storages​

In Crawlee v1.0, the Dataset, KeyValueStore, and RequestQueue storage APIs have been updated for consistency and simplicity. Below is a detailed overview of what's new, what's changed, and what's been removed.

See the Storages guide for more details.

Dataset​

The Dataset API now includes several new methods, such as:

  • get_metadata - retrieves metadata information for the dataset.
  • purge - completely clears the dataset, including all items (keeps the metadata only).
  • list_items - returns the dataset's items in a list format.

Some older methods have been removed or replaced:

  • from_storage_object constructor has been removed. You should now use the open method with either a name or id parameter.
  • get_info method and the storage_object property have been replaced by the new get_metadata method.
  • set_metadata method has been removed.
  • write_to_json and write_to_csv methods have been removed; instead, use the export_to method for exporting data in different formats.

Key-value store​

The KeyValueStore API now includes several new methods, such as:

  • get_metadata - retrieves metadata information for the key-value store.
  • purge - completely clears the key-value store, removing all keys and values (keeps the metadata only).
  • delete_value - deletes a specific key and its associated value.
  • list_keys - lists all keys in the key-value store.

Some older methods have been removed or replaced:

  • from_storage_object - removed; use the open method with either a name or id instead.
  • get_info and storage_object - replaced by the new get_metadata method.
  • set_metadata method has been removed.

Request queue​

The RequestQueue API now includes several new methods, such as:

  • get_metadata - retrieves metadata information for the request queue.
  • purge - completely clears the request queue, including all pending and processed requests (keeps the metadata only).
  • add_requests - replaces the previous add_requests_batched method, offering the same functionality under a simpler name.

Some older methods have been removed or replaced:

  • from_storage_object - removed; use the open method with either a name or id instead.
  • get_info and storage_object - replaced by the new get_metadata method.
  • get_request has argument unique_key instead of request_id as the id field was removed from the Request.
  • set_metadata method has been removed.

Some changes in the related model classes:

  • resource_directory in RequestQueueMetadata - removed; use the corresponding path_to_* property instead.
  • stats field in RequestQueueMetadata - removed as it was unused.
  • RequestQueueHead - replaced by RequestQueueHeadWithLocks.

New architecture of storage clients​

In v1.0, the storage client system has been completely reworked to simplify implementation and make custom storage clients easier to write.

See the Storage clients guide for more details.

New dedicated storage clients​

Previously, MemoryStorageClient handled both in-memory storage and optional file system persistence. This has now been split into two distinct storage clients:

  • MemoryStorageClient - Stores all data in memory only.
  • FileSystemStorageClient - Persists data on the file system, with in-memory caching for better performance.

Before (v0.6):

from crawlee.configuration import Configuration
from crawlee.storage_clients import MemoryStorageClient

# In-memory only
configuration = Configuration(persist_storage=False)
storage_client = MemoryStorageClient.from_config(configuration)

# File-system persistence
configuration = Configuration(persist_storage=True)
storage_client = MemoryStorageClient.from_config(configuration)

Now (v1.0):

from crawlee.storage_clients import MemoryStorageClient, FileSystemStorageClient

# In-memory only
storage_client = MemoryStorageClient()

# File-system persistence
storage_client = FileSystemStorageClient()

Registering a storage client​

The way you register a storage client remains unchanged:

from crawlee import service_locator
from crawlee.crawlers import ParselCrawler
from crawlee.storage_clients import MemoryStorageClient
from crawlee.storages import Dataset

# Create custom storage client
storage_client = MemoryStorageClient()

# Then register it globally
service_locator.set_storage_client(storage_client)

# Or use it for a single crawler only
crawler = ParselCrawler(storage_client=storage_client)

# Or use it for a single storage only
dataset = await Dataset.open(
    name='my_dataset',
    storage_client=storage_client,
)

Instance caching​

Instance caching of Dataset.open, KeyValueStore.open, and RequestQueue.open now return the same instance for the same arguments. Direct calls to StorageClient.open_* always return new instances.

Writing custom storage clients​

The interface for custom storage clients has been simplified:

  • One storage client per storage type (RequestQueue, KeyValueStore, Dataset).
  • Collection storage clients have been removed.
  • The number of methods that have to be implemented have been reduced.

Other smaller updates​

There are more smaller updates.

Python version support​

We drop support for Python 3.9. The minimum supported version is now Python 3.10.

Changes in Configuration​

The fields persist_storage and persist_metadata have been removed from the Configuration. Persistence is now determined only by which storage client class you use.

Changes in Request​

Request objects no longer have id field and all its usages have been transferred to unique_key field.

Changes in HttpResponse​

The method HttpResponse.read is now asynchronous. This affects all HTTP-based crawlers.

Before (v0.6):

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext

async def main() -> None:
    crawler = ParselCrawler()

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        content = context.http_response.read()
        # ...

    await crawler.run(['https://crawlee.dev/'])

Now (v1.0):

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext

async def main() -> None:
    crawler = ParselCrawler()

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        content = await context.http_response.read()
        # ...

    await crawler.run(['https://crawlee.dev/'])