Version: Next

Scaling

How crawlers adjust concurrency to available resources and limit the request rate per domain.

📄️Scaling crawlers

Learn how to scale your crawlers by controlling concurrency and limiting requests per minute.

How to throttle requests per domain using the ThrottlingRequestManager.