Version: Next

Playwright crawler

A PlaywrightCrawler is a browser-based crawler. In contrast to HTTP-based crawlers like ParselCrawler or BeautifulSoupCrawler, it uses a real browser to render pages and extract data. It is built on top of the Playwright browser automation library. While browser-based crawlers are typically slower and less efficient than HTTP-based crawlers, they can handle dynamic, client-side rendered sites that standard HTTP-based crawlers cannot manage.

When to use Playwright crawler

Use PlaywrightCrawler in scenarios that require full browser capabilities, such as:

Dynamic content rendering: Required when pages rely on heavy JavaScript to load or modify content in the browser.
Anti-scraping protection: Helpful for sites using JavaScript-based security or advanced anti-automation measures.
Complex cookie management: Necessary for sites with session or cookie requirements that standard HTTP-based crawlers cannot handle easily.

If HTTP-based crawlers are insufficient, PlaywrightCrawler can address these challenges.

Basic example

The following example crawls the Hacker News website using Chromium. The crawler manages the browser and page instances for you. In the request handler, Playwright's API is used to extract the title, rank, and URL of each post, and links to the next pages are enqueued to keep the crawl going.

Run on

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
    PlaywrightPreNavCrawlingContext,
)


async def main() -> None:
    crawler = PlaywrightCrawler(
        # Limit the crawl to max requests. Remove or increase it for crawling all links.
        max_requests_per_crawl=10,
        # Headless mode, set to False to see the browser in action.
        headless=False,
        # Browser types supported by Playwright.
        browser_type='chromium',
    )

    # Define the default request handler, which will be called for every request.
    # The handler receives a context parameter, providing various properties and
    # helper methods. Here are a few key ones we use for demonstration:
    # - request: an instance of the Request class containing details such as the URL
    #   being crawled and the HTTP method used.
    # - page: Playwright's Page object, which allows interaction with the web page
    #   (see https://playwright.dev/python/docs/api/class-page for more details).
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        # Extract data from the page using Playwright's API.
        posts = await context.page.query_selector_all('.athing')
        data = []

        for post in posts:
            # Get the HTML elements for the title and rank within each post.
            title_element = await post.query_selector('.title a')
            rank_element = await post.query_selector('.rank')

            # Extract the data we want from the elements.
            title = await title_element.inner_text() if title_element else None
            rank = await rank_element.inner_text() if rank_element else None
            href = await title_element.get_attribute('href') if title_element else None

            data.append({'title': title, 'rank': rank, 'href': href})

        # Push the extracted data to the default dataset. In local configuration,
        # the data will be stored as JSON files in ./storage/datasets/default.
        await context.push_data(data)

        # Find a link to the next page and enqueue it if it exists.
        await context.enqueue_links(selector='.morelink')

    # Define a hook that will be called each time before navigating to a new URL.
    # The hook receives a context parameter, providing access to the request and
    # browser page among other things. In this example, we log the URL being
    # navigated to.
    @crawler.pre_navigation_hook
    async def log_navigation_url(context: PlaywrightPreNavCrawlingContext) -> None:
        context.log.info(f'Navigating to {context.request.url} ...')

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://news.ycombinator.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Advanced configuration

The PlaywrightCrawler uses other Crawlee components under the hood, notably BrowserPool and PlaywrightBrowserPlugin. These components let you to configure the browser and context settings, launch multiple browsers, and apply pre-navigation hooks. You can create your own instances of these components and pass them to the PlaywrightCrawler constructor.

The PlaywrightBrowserPlugin manages how browsers are launched and how browser contexts are created. It accepts browser launch and new context options.
The BrowserPool manages the lifecycle of browser instances (launching, recycling, etc.). You can customize its behavior to suit your needs.

Managing multiple browsers

The BrowserPool allows you to manage multiple browsers. Each browser instance is managed by a separate PlaywrightBrowserPlugin and can be configured independently. This is useful for scenarios like testing multiple configurations or implementing browser rotation to help avoid blocks or detect different site behaviors.

Run on

import asyncio

from crawlee.browsers import BrowserPool, PlaywrightBrowserPlugin
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # Create a plugin for each required browser.
    plugin_chromium = PlaywrightBrowserPlugin(
        browser_type='chromium', max_open_pages_per_browser=1
    )
    plugin_firefox = PlaywrightBrowserPlugin(
        browser_type='firefox', max_open_pages_per_browser=1
    )

    crawler = PlaywrightCrawler(
        browser_pool=BrowserPool(plugins=[plugin_chromium, plugin_firefox]),
        # Limit the crawl to max requests. Remove or increase it for crawling all links.
        max_requests_per_crawl=10,
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        browser_name = (
            context.page.context.browser.browser_type.name
            if context.page.context.browser
            else 'undefined'
        )
        context.log.info(f'Processing {context.request.url} with {browser_name} ...')

        await context.enqueue_links()

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://crawlee.dev', 'https://apify.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Browser launch and context configuration

The PlaywrightBrowserPlugin provides access to all relevant Playwright configuration options for both browser launches and new browser contexts. You can specify these options in the constructor of PlaywrightBrowserPlugin or PlaywrightCrawler:

Run on

import asyncio

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    crawler = PlaywrightCrawler(
        headless=False,
        browser_type='chromium',
        # Browser launch options
        browser_launch_options={
            # For support `msedge` channel you need to install it
            # `playwright install msedge`
            'channel': 'msedge',
            'slow_mo': 200,
        },
        # Context launch options, applied to each page as it is created
        browser_new_context_options={
            'color_scheme': 'dark',
            # Set headers
            'extra_http_headers': {
                'Custom-Header': 'my-header',
                'Accept-Language': 'en',
            },
            # Set only User Agent
            'user_agent': 'My-User-Agent',
        },
        # Limit the crawl to max requests. Remove or increase it for crawling all links.
        max_requests_per_crawl=10,
    )

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        await context.enqueue_links()

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

You can also configure each plugin used by BrowserPool:

import asyncio

from crawlee.browsers import BrowserPool, PlaywrightBrowserPlugin
from crawlee.crawlers import PlaywrightCrawler


async def main() -> None:
    crawler = PlaywrightCrawler(
        browser_pool=BrowserPool(
            plugins=[
                PlaywrightBrowserPlugin(
                    browser_type='chromium',
                    browser_launch_options={
                        'headless': False,
                        'channel': 'msedge',
                        'slow_mo': 200,
                    },
                    browser_new_context_options={
                        'color_scheme': 'dark',
                        'extra_http_headers': {
                            'Custom-Header': 'my-header',
                            'Accept-Language': 'en',
                        },
                        'user_agent': 'My-User-Agent',
                    },
                )
            ]
        )
    )

    # ...


if __name__ == '__main__':
    asyncio.run(main())

Extending the browser plugin

Use the configuration options provided by PlaywrightBrowserPlugin when the standard Playwright launch API supports the required browser. This keeps browser launch and context settings, incognito behavior, and user data directory handling on the built-in path without requiring a custom plugin.

Create a subclass only when an integration uses a different browser launch API. Override new_browser() to launch the browser and return a PlaywrightBrowserController. BrowserPool initializes the plugin before calling this method, forwards its browser context options when creating pages, and manages the returned controller's lifecycle.

Keep the inherited plugin configuration intact so that browser context options remain available to BrowserPool. When constructing the controller, explicitly preserve or intentionally replace use_incognito_pages, max_open_pages_per_browser, and fingerprint or header generation behavior. The custom launch path must likewise define how it applies browser launch options and user_data_dir.

See the Camoufox example for a complete implementation that uses a different browser launcher.

Browser pool lifecycle hooks

The BrowserPool exposes lifecycle hooks for both browser launches and page creation/closure. To use them, create a BrowserPool instance and pass it to PlaywrightCrawler via the browser_pool argument.

Browser launch hooks

The pre_launch_hook and post_launch_hook are called once per browser instance, before and after it is launched. Use them for logging, metrics, or any setup at the browser level. Note that these hooks are not called when a new page is created in an already-running browser.

Run on

from __future__ import annotations

import asyncio
import logging
from typing import TYPE_CHECKING

from crawlee.browsers import BrowserPool
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

if TYPE_CHECKING:
    from crawlee.browsers._browser_controller import BrowserController
    from crawlee.browsers._browser_plugin import BrowserPlugin

logger = logging.getLogger(__name__)


async def main() -> None:
    async with BrowserPool() as browser_pool:

        @browser_pool.pre_launch_hook
        async def log_browser_launch(page_id: str, plugin: BrowserPlugin) -> None:
            """Log before a new browser instance is launched."""
            logger.info(f'Launching {plugin.browser_type} browser for page {page_id}...')

        @browser_pool.post_launch_hook
        async def log_browser_launched(
            page_id: str, controller: BrowserController
        ) -> None:
            """Log after a new browser instance has been launched."""
            logger.info(f'Browser launched for page {page_id}, controller: {controller}')

        crawler = PlaywrightCrawler(
            browser_pool=browser_pool,
            max_requests_per_crawl=5,
        )

        @crawler.router.default_handler
        async def request_handler(context: PlaywrightCrawlingContext) -> None:
            context.log.info(f'Processing {context.request.url} ...')

            await context.enqueue_links()

        # Run the crawler with the initial list of URLs.
        await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Page lifecycle hooks

For additional setup or event-driven actions around page creation and closure, the BrowserPool exposes four hooks: pre_page_create_hook, post_page_create_hook, pre_page_close_hook, and post_page_close_hook.

Run on

from __future__ import annotations

import asyncio
import logging
from typing import TYPE_CHECKING, Any

from crawlee.browsers import BrowserPool
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.storages import KeyValueStore

if TYPE_CHECKING:
    from crawlee.browsers._browser_controller import BrowserController
    from crawlee.browsers._types import CrawleePage
    from crawlee.proxy_configuration import ProxyInfo

logger = logging.getLogger(__name__)


async def main() -> None:
    async with BrowserPool() as browser_pool:

        @browser_pool.pre_page_create_hook
        async def log_page_init(
            page_id: str,
            _browser_controller: BrowserController,
            _browser_new_context_options: dict[str, Any],
            _proxy_info: ProxyInfo | None,
        ) -> None:
            """Log when a new page is about to be created."""
            logger.info(f'Creating page {page_id}...')

        @browser_pool.post_page_create_hook
        async def set_viewport(
            crawlee_page: CrawleePage, _browser_controller: BrowserController
        ) -> None:
            """Set a fixed viewport size on each newly created page."""
            await crawlee_page.page.set_viewport_size({'width': 1280, 'height': 1024})

        @browser_pool.pre_page_close_hook
        async def save_screenshot(
            crawlee_page: CrawleePage, _browser_controller: BrowserController
        ) -> None:
            """Save a screenshot to KeyValueStore before each page is closed."""
            kvs = await KeyValueStore.open()

            screenshot = await crawlee_page.page.screenshot()
            await kvs.set_value(
                key=f'screenshot-{crawlee_page.id}',
                value=screenshot,
                content_type='image/png',
            )
            logger.info(f'Saved screenshot for page {crawlee_page.id}.')

        @browser_pool.post_page_close_hook
        async def log_page_closed(
            page_id: str, _browser_controller: BrowserController
        ) -> None:
            """Log after each page is closed."""
            logger.info(f'Page {page_id} closed successfully.')

        crawler = PlaywrightCrawler(
            browser_pool=browser_pool,
            max_requests_per_crawl=5,
        )

        @crawler.router.default_handler
        async def request_handler(context: PlaywrightCrawlingContext) -> None:
            context.log.info(f'Processing {context.request.url} ...')

            await context.enqueue_links()

        # Run the crawler with the initial list of URLs.
        await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Navigation hooks allow for additional configuration at specific points during page navigation. The pre_navigation_hook is called before each navigation and provides PlaywrightPreNavCrawlingContext - including the page instance and a block_requests helper for filtering unwanted resource types and URL patterns, covered in Blocking network requests. Similarly, the post_navigation_hook is called after each navigation and provides PlaywrightPostNavCrawlingContext - useful for post-load checks such as detecting CAPTCHAs or verifying page state.

Run on

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
    PlaywrightPostNavCrawlingContext,
    PlaywrightPreNavCrawlingContext,
)
from crawlee.errors import SessionError


async def main() -> None:
    crawler = PlaywrightCrawler(max_requests_per_crawl=10)

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        await context.enqueue_links()

    @crawler.pre_navigation_hook
    async def configure_page(context: PlaywrightPreNavCrawlingContext) -> None:
        context.log.info(f'Navigating to {context.request.url} ...')

        # block stylesheets, images, fonts and other static assets
        # to speed up page loading
        await context.block_requests()

    @crawler.post_navigation_hook
    async def custom_captcha_check(context: PlaywrightPostNavCrawlingContext) -> None:
        # check if the page contains a captcha
        captcha_element = context.page.locator('input[name="captcha"]').first
        if await captcha_element.is_visible():
            context.log.warning('Captcha detected! Skipping the page.')
            raise SessionError('Captcha detected')

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Blocking network requests

Loading non-essential resources like images, styles, or analytics scripts wastes bandwidth and slows down crawling. The block_requests helper provides the most efficient way to block such requests, as it operates directly in the browser. By default, it blocks URLs matching the following patterns:

['.css', '.webp', '.jpg', '.jpeg', '.png', '.svg', '.gif', '.woff', '.pdf', '.zip']

You can replace the default patterns with your own by providing url_patterns, or extend them by passing additional patterns in extra_url_patterns.

Run on

import asyncio

from crawlee.crawlers import (
    PlaywrightCrawler,
    PlaywrightCrawlingContext,
    PlaywrightPreNavCrawlingContext,
)


async def main() -> None:
    crawler = PlaywrightCrawler(
        # Limit the crawl to max requests. Remove or increase it for crawling all links.
        max_requests_per_crawl=10,
    )

    # Define the default request handler, which will be called for every request.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        await context.enqueue_links()

    # Define the hook, which will be called before every request.
    @crawler.pre_navigation_hook
    async def navigation_hook(context: PlaywrightPreNavCrawlingContext) -> None:
        context.log.info(f'Navigating to {context.request.url} ...')

        # Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
        await context.block_requests(extra_url_patterns=['adsbygoogle.js'])

    # Run the crawler with the initial list of URLs.
    await crawler.run(['https://crawlee.dev/'])


if __name__ == '__main__':
    asyncio.run(main())

Capturing screenshots

Playwright's page.screenshot() method makes it easy to capture screenshots of the pages your crawler visits. The following example stores each screenshot in the key-value store as a PNG image, under a unique key generated from the page URL.

Run on

import asyncio

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.storages import KeyValueStore


async def main() -> None:
    crawler = PlaywrightCrawler(
        # Limit the crawl to max requests. Remove or increase it for crawling all links.
        max_requests_per_crawl=10,
        # Headless mode, set to False to see the browser in action.
        headless=False,
        # Browser types supported by Playwright.
        browser_type='chromium',
    )

    # Open the default key-value store.
    kvs = await KeyValueStore.open()

    # Define the default request handler, which will be called for every request.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')

        # Capture the screenshot of the page using Playwright's API.
        screenshot = await context.page.screenshot()
        name = context.request.url.split('/')[-1]

        # Store the screenshot in the key-value store.
        await kvs.set_value(
            key=f'screenshot-{name}',
            value=screenshot,
            content_type='image/png',
        )

    # Run the crawler with the initial list of URLs.
    await crawler.run(
        [
            'https://crawlee.dev',
            'https://apify.com',
            'https://example.com',
        ]
    )


if __name__ == '__main__':
    asyncio.run(main())

Playwright crawler

When to use Playwright crawler​

Basic example​

Advanced configuration​

Managing multiple browsers​

Browser launch and context configuration​

Extending the browser plugin​

Browser pool lifecycle hooks​

Browser launch hooks​

Page lifecycle hooks​

Navigation hooks​

Blocking network requests​

Capturing screenshots​

When to use Playwright crawler

Basic example

Advanced configuration

Managing multiple browsers

Browser launch and context configuration

Extending the browser plugin

Browser pool lifecycle hooks

Browser launch hooks

Page lifecycle hooks

Navigation hooks

Blocking network requests

Capturing screenshots