Skip to main content

Playwright crawler

This example demonstrates how to use PlaywrightCrawler to recursively scrape the Hacker news website using headless Chromium and Playwright.

The PlaywrightCrawler manages the browser and page instances, simplifying the process of interacting with web pages. In the request handler, Playwright's API is used to extract data from each post on the page. Specifically, it retrieves the title, rank, and URL of each post. Additionally, the handler enqueues links to the next pages to ensure continuous scraping. This setup is ideal for scraping dynamic web pages where JavaScript execution is required to render the content.

import asyncio

from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext

async def main() -> None:
crawler = PlaywrightCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
# Headless mode, set to False to see the browser in action.
# Browser types supported by Playwright.

# Define the default request handler, which will be called for every request.
# The handler receives a context parameter, providing various properties and
# helper methods. Here are a few key ones we use for demonstration:
# - request: an instance of the Request class containing details such as the URL
# being crawled and the HTTP method used.
# - page: Playwright's Page object, which allows interaction with the web page
# (see for more details).
async def request_handler(context: PlaywrightCrawlingContext) -> None:'Processing {context.request.url} ...')

# Extract data from the page using Playwright's API.
posts = await'.athing')
data = []

for post in posts:
# Get the HTML elements for the title and rank within each post.
title_element = await post.query_selector('.title a')
rank_element = await post.query_selector('.rank')

# Extract the data we want from the elements.
title = await title_element.inner_text() if title_element else None
rank = await rank_element.inner_text() if rank_element else None
href = await title_element.get_attribute('href') if title_element else None

data.append({'title': title, 'rank': rank, 'href': href})

# Push the extracted data to the default dataset. In local configuration,
# the data will be stored as JSON files in ./storage/datasets/default.
await context.push_data(data)

# Find a link to the next page and enqueue it if it exists.
await context.enqueue_links(selector='.morelink')

# Run the crawler with the initial list of URLs.

if __name__ == '__main__':