Crawling links
How to crawl a list of URLs, discover and enqueue new links on the fly, and control which links get followed.
Filling and submitting web forms
How to inspect a web form, prepare a POST request from its fields, and submit it with an HTTP crawler.
Logging in with a crawler
How to log in to websites with Crawlee.
Avoid getting blocked
How to avoid getting blocked by anti-bot protections, using browser fingerprints and stealth browsers.
Respecting robots.txt
How to make a crawler respect the rules websites declare for bots in their robots.txt file.
Stopping and resuming crawlers
How to stop a crawler from a request handler, keep it alive while waiting for more requests, and resume an interrupted crawl.
Running crawlers in parallel
How to run two crawlers in parallel and pass links between them through request queues.
Running in web server
How to run a crawler inside a web server and return crawl results in response to HTTP requests.
Trace and monitor crawlers
Learn how to instrument your crawlers with OpenTelemetry to trace request handling, identify bottlenecks, monitor performance, and visualize telemetry data using Jaeger for performance optimization.
Creating web archive
How to create a Web ARChive (WARC) of the crawled pages with Crawlee.
Security of web scraping
Understand the security risks of running web crawlers and the controls Crawlee applies by default.
Migrating from Scrapy
A guide for Scrapy users on how to move an existing project to Crawlee for Python, mapping Scrapy concepts to their Crawlee equivalents.