Crawling context

Every request handler receives a single argument, the crawling context. It holds the request being processed, the session and proxy used for it, and a set of helper functions for storing data and adding new requests. The crawler creates a new context for each request, so handlers don't share it.

Context types

All contexts extend BasicCrawlingContext, which is what BasicCrawler passes to its handlers. Each crawler builds on it and adds the attributes that make sense for the way it fetches pages:

• HttpCrawlingContext adds http_response, the response returned by the HTTP client.
• ParsedHttpCrawlingContext adds parsed_content together with the enqueue_links and extract_links helpers. HttpCrawler uses it with the raw response body as parsed_content.
• BeautifulSoupCrawlingContext and ParselCrawlingContext are parsed contexts that expose the parsed document as soup and selector.
• PlaywrightPreNavCrawlingContext is what Playwright pre-navigation hooks receive. It adds the Playwright page before navigation, goto_options and the block_requests helper.
• PlaywrightCrawlingContext is what PlaywrightCrawler handlers receive. It adds the navigation response, enqueue_links, extract_links and infinite_scroll.

Pre-navigation hooks of HTTP crawlers receive a plain BasicCrawlingContext, because no response exists yet. Their post-navigation hooks receive an HttpCrawlingContext. Annotate your handler with the context type of your crawler to get the right attributes and type checking.

For details on the crawler-specific attributes, see HTTP crawlers and Playwright crawler.

Request, session and proxy

The request attribute is the Request being processed. It carries the URL, the label used for routing, user data and the retry count. For details, see Requests.

The session attribute is the Session assigned to the request, or None when the crawler runs without a session pool. You can call mark_bad() or retire() on it from the handler when a page shows signs of blocking. For details, see Session management.

The proxy_info attribute is a ProxyInfo describing the proxy used for the request, or None when no proxy is configured. For details, see Proxy management.

The log attribute is the crawler's logger, so messages from handlers follow the same log configuration as the crawler's own. For details, see Logging.

Context helpers

The context provides helpers that operate on the crawler's storages and request queue:

• push_data stores one item or a list of items in a dataset. It writes to the default dataset unless you pass dataset_id, dataset_name or dataset_alias.
• add_requests adds URLs or Request objects to the request queue. Relative URLs resolve against the URL of the current page, and each new request gets a crawl depth one higher than the current one.
• enqueue_links extracts links from the current page and adds them to the request queue through add_requests. It's available on the parsed HTTP contexts and on PlaywrightCrawlingContext. For filtering links and choosing an enqueue strategy, see Crawling links.
• get_key_value_store opens a key-value store, the default one unless you pass id, name or alias.
• use_state returns a dictionary shared by all handler calls of the crawler. The crawler keeps it in the default key-value store and saves it automatically. This helper is experimental and its interface may change.
• send_request sends an extra HTTP request through the crawler's HTTP client, with the session and proxy of the current request. Its response isn't parsed, and the request doesn't go through the request queue.

The following example uses most of these helpers in a ParselCrawler handler and reads the results back from the crawler once the run finishes.

Run on

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=5)

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        # The request being processed and the crawler's logger.
        context.log.info(f'Processing {context.request.url} ...')

        # Store a result in the default dataset.
        await context.push_data(
            {
                'url': context.request.url,
                'title': context.selector.css('title::text').get(),
            }
        )

        # Store a value in the default key-value store.
        kvs = await context.get_key_value_store()
        await kvs.set_value('last-url', context.request.url)

        # Keep a counter in the crawler state shared by all handler calls.
        state = await context.use_state({'pages': 0})
        pages = state['pages']
        if isinstance(pages, int):
            state['pages'] = pages + 1

        # Add links found on the page to the request queue.
        await context.enqueue_links()

    await crawler.run(['https://crawlee.dev/'])

    kvs = await crawler.get_key_value_store()
    print(f'Last URL: {await kvs.get_value("last-url")}')

    state = await crawler.use_state()
    print(f'Pages counted in state: {state["pages"]}')

    data = await crawler.get_data()
    print(f'Items in dataset: {len(data.items)}')


if __name__ == '__main__':
    asyncio.run(main())

Results are committed after the handler succeeds

The push_data, add_requests and enqueue_links calls and the set_value calls on a key-value store from get_key_value_store don't write anything right away. The context records them in an internal RequestHandlerRunResult object, and the crawler commits the recorded changes only after the request handler returns without an exception. Reading a key from that key-value store in the same handler returns the value you've set, even before the commit.

When the handler raises, the crawler discards the recorded changes and retries the request. A handler that pushes an item and then fails on the next line doesn't leave a partial or duplicate item in the dataset. The retry starts with an empty record and produces the output once, when it succeeds. The same applies to requests added by a failed attempt.

The example simulates a failure on the first attempt. The dataset ends up with a single item, the one from the second attempt.

Run on

import asyncio

from crawlee.crawlers import BasicCrawler, BasicCrawlingContext


async def main() -> None:
    crawler = BasicCrawler(max_request_retries=2)

    @crawler.router.default_handler
    async def request_handler(context: BasicCrawlingContext) -> None:
        attempt = context.request.retry_count + 1
        context.log.info(f'Attempt {attempt} for {context.request.url}')

        # Buffered until the handler finishes without an exception.
        await context.push_data({'url': context.request.url, 'attempt': attempt})

        # Fail on the first attempt, so the crawler retries the request.
        if attempt == 1:
            raise RuntimeError('Simulated failure')

    await crawler.run(['https://crawlee.dev/'])

    # Only the item from the successful attempt is stored.
    data = await crawler.get_data()
    print(data.items)


if __name__ == '__main__':
    asyncio.run(main())

A few parts of the context write directly, outside of the buffer:

• Changes to the dictionary from use_state are visible to other handlers immediately and stay in place when the handler fails.
• Requests made with send_request happen when you call it.
• Storages you open yourself, for example with Dataset.open, write immediately.

Error handlers and failed request handlers receive a copy of the context whose push_data, add_requests and get_key_value_store write directly, because the request has already failed and there's no successful run to wait for. For details, see Error handling.