Fundamentals
How a crawler processes requests: the architecture, the request object, the crawling context, routing, and error handling.
Crawlers
The crawler classes for HTTP, browser, adaptive, and LLM-based crawling, and when to use each one.
Storage
Where crawlers keep data and requests: datasets, key-value stores, request queues, storage backends, and request loaders.
HTTP and networking
How crawlers send requests over the network: HTTP clients, headers, sessions, cookies, and proxies.
Scaling
How crawlers adjust concurrency to available resources and limit the request rate per domain.
Runtime
How to configure a crawler run and observe it through events, statistics, and logs.
Advanced
The service locator and the extension points for plugging your own components into Crawlee.