Request Storage
How to store the requests your crawler will go through
Request loaders
How to manage the requests your crawler will go through.
Result Storage
Where are you going to store all of that juicy scraped data?!
HTTP clients
Learn about Crawlee's HTTP client architecture, how to switch between different implementations, and create custom HTTP clients for specialized web scraping needs.
Configuration
Configuring Crawlee parameters
CheerioCrawler
Your first steps into the world of scraping with Crawlee
JavaScript rendering
Your first steps into the world of scraping with Crawlee
Adaptive crawling
One request handler, a browser only where the page needs it
Remote browsers
Connect Crawlee crawlers to remote browser services like Browser Use, Browserbase, Browserless, or Steel.
Proxy Management
Using proxies to get around those annoying IP-blocks
Session Management
How to manage your cookies, proxy IP rotations and more
Scaling our crawlers
To infinity and beyond! ...within limits
Running multiple crawlers
Separate queues, feeding one crawler from another, and knowing when to stop
Trace and monitor crawlers
How to use OpenTelemetry to trace and monitor your crawlers
Avoid getting blocked
How to avoid getting blocked when scraping
Cookie modals
Ways to get past cookie consent banners in a Crawlee crawler.
Impit HTTP Client
Browser impersonation for HTTP requests using the Impit library
Got Scraping
Blazing fast cURL alternative for modern web scraping
TypeScript Projects
Stricter, safer, and better development experience
Public API
What Crawlee promises not to break, and what it doesn't
Running in Docker
Example Docker images to run your crawlers
StagehandCrawler
AI-powered web crawling with natural language browser automation
Running in web server
Run Crawlee in web server using a request/response approach
Parallel Scraping
Parallelizing your scrapers with Crawlee
Custom statistics fields
Track your own counters alongside the built-in crawler statistics
Request lifecycle
What happens to a request between the queue and the request handler, and what happens when it fails
Customizing the crawling context
Extend the crawling context ad hoc or build custom crawlers on top of ContextPipeline
Using a custom HTTP client (Experimental)
Use a custom HTTP client for `sendRequest` and plain-HTTP crawling
Custom logger
Use your own logging library (Winston, Pino, etc.) with Crawlee