Skip to main content

Crawl multiple URLs

This example demonstrates how to crawl a specified list of URLs using different crawlers. You'll learn how to set up the crawler, define a request handler, and run the crawler with multiple URLs. This setup is useful for scraping data from multiple pages or websites concurrently.

import asyncio

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main() -> None:
crawler = BeautifulSoupCrawler()

# Define the default request handler, which will be called for every request.
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:'Processing {context.request.url} ...')

# Run the crawler with the initial list of requests.

if __name__ == '__main__':