Stagehand crawler

A StagehandCrawler extends PlaywrightCrawler with AI-powered browser automation via Stagehand. Instead of writing CSS selectors or XPath expressions, you describe what you want in plain English and the AI model takes care of the rest.

Each page in the crawling context is a StagehandPage - a drop-in replacement for the standard Playwright Page that adds four AI methods:

• page.act(**kwargs) - perform an action using a natural language instruction
• page.extract(**kwargs) - extract structured data from the page using AI
• page.observe(**kwargs) - get a list of AI-suggested actions available on the page
• page.execute(**kwargs) - run an autonomous multi-step agent on the page

All standard Playwright methods remain available alongside these AI methods.

When to use StagehandCrawler

Use StagehandCrawler when:

• Selectors are brittle or unknown - the AI can locate elements by their visual role or label rather than a specific CSS class.
• Interactions are complex - multi-step forms, dynamic menus, or context-dependent flows that are hard to script.
• Rapid prototyping - you want to build a scraper quickly without spending time reverse-engineering the page structure.

For straightforward scraping tasks where the page structure is stable and well-known, PlaywrightCrawler is more efficient, read more in that guide.

Installation

StagehandCrawler requires the stagehand optional dependency group:

pip install 'crawlee[stagehand]'

or with uv:

uv add 'crawlee[stagehand]'

Basic usage

The example below demonstrates the typical usage pattern: dismiss cookie banners with act() and extract structured data with extract().

import asyncio
from typing import cast

from crawlee.browsers import StagehandOptions
from crawlee.crawlers import StagehandCrawler, StagehandCrawlingContext


async def main() -> None:
    crawler = StagehandCrawler(
        stagehand_options=StagehandOptions(
            model_api_key='your-openai-api-key',
            model='openai/gpt-5.4-nano',
        ),
        max_requests_per_crawl=5,
    )

    @crawler.router.default_handler
    async def handler(context: StagehandCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        # Dismiss overlays or interact with the page using natural language.
        await context.page.act(input='Click the accept cookies button if present')

        # Extract data from the page using AI.
        extracted = await context.page.extract(
            instruction='Get the page title and the main heading text',
            schema={
                'type': 'object',
                'properties': {
                    'title': {'type': 'string'},
                    'heading': {'type': 'string'},
                },
            },
        )

        extract_result = extracted.data.result

        if isinstance(extract_result, dict):
            # Push extracted data to the dataset
            # Use `cast()` to provide a more specific type hint for the extracted data.
            await context.push_data(cast('dict[str, str | None]', extract_result))

    await crawler.run(['https://example.com'])


if __name__ == '__main__':
    asyncio.run(main())

StagehandOptions configuration

Stagehand-specific settings are provided via StagehandOptions. Pass the instance to the stagehand_options argument of StagehandCrawler.

AI page operations

act - perform actions

Use act() to interact with the page using a natural language instruction:

await context.page.act(input='Click the "Sign in" button')

extract - structured data extraction

Use extract() to pull structured data from the page. Pass a JSON Schema via schema to enforce the output shape:

data = await context.page.extract(
    instruction='Extract the top comment on this page',
    schema={
        'type': 'object',
        'properties': {
            'comment_text': {'type': 'string'},
            'author': {'type': 'string'},
        },
        'required': ['comment_text'],
    },
)

observe - inspect available actions

Use observe() to get AI-suggested actions currently available on the page. Useful for debugging or building adaptive workflows:

actions = await context.page.observe(
    instruction='What actions are available in the navigation menu?'
)

execute - autonomous multi-step agent

Use execute() for longer autonomous tasks that span multiple interactions:

result = await context.page.execute(
    agent_config={},
    execute_options={
        'instruction': 'Search for "web scraping" and return the titles of the first five results',
    })

Browserbase integration

By default, Stagehand launches a local Chromium browser. To use Browserbase - a managed cloud browser service - set env='BROWSERBASE' in StagehandOptions and supply the required credentials:

import asyncio
from typing import cast

from crawlee.browsers import StagehandOptions
from crawlee.crawlers import StagehandCrawler, StagehandCrawlingContext


async def main() -> None:
    # Use Browserbase cloud browser instead of a local Chromium instance.
    crawler = StagehandCrawler(
        stagehand_options=StagehandOptions(
            env='BROWSERBASE',
            browserbase_api_key='your-browserbase-api-key',
            project_id='your-project-id',
            model_api_key='your-openai-api-key',
            model='openai/gpt-5.4-nano',
        ),
        max_requests_per_crawl=5,
    )

    @crawler.router.default_handler
    async def handler(context: StagehandCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        extracted = await context.page.extract(
            instruction='Get the main content of the page',
        )

        extract_result = extracted.data.result

        await context.push_data(cast('dict[str, str | None]', extract_result))

    await crawler.run(['https://example.com'])


if __name__ == '__main__':
    asyncio.run(main())

Browserbase credentials (browserbase_api_key, project_id) can also be provided via the BROWSERBASE_API_KEY and BROWSERBASE_PROJECT_ID environment variables.

Browser configuration limitations

Because Stagehand manages the browser session internally via CDP, only Chromium is supported. Browser settings are limited to the subset accepted by Stagehand's BrowserLaunchOptions - headless, args, viewport, proxy, locale, executable_path, and a few others. Features like full browser fingerprinting (canvas, WebGL, screen properties) and incognito pages are not supported. Fingerprint-consistent HTTP headers (User-Agent, Accept, sec-ch-ua) are still injected automatically.

Conclusion

This guide introduced StagehandCrawler and its AI page operations: act(), extract(), observe(), and execute(). You learned how to configure Stagehand via StagehandOptions and switch to Browserbase for cloud browser execution. If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!