Helper ClassesBasicCrawlerOptionsOn this pageBasicCrawlerOptions Copy of the parameter types of BasicCrawler.__init__ meant for typing forwarded init args in subclasses.Index Propertiesconcurrency_settingsconfigurationconfigure_loggingevent_managerhttp_clientmax_request_retriesmax_requests_per_crawlmax_session_rotationsproxy_configurationrequest_handlerrequest_handler_timeoutrequest_providerretry_on_blockedsession_poolstatisticsuse_session_poolProperties concurrency_settingsconcurrency_settings: NotRequired[ConcurrencySettings]configurationconfiguration: NotRequired[Configuration]configure_loggingconfigure_logging: NotRequired[bool]event_managerevent_manager: NotRequired[EventManager]http_clienthttp_client: NotRequired[BaseHttpClient]max_request_retriesmax_request_retries: NotRequired[int]max_requests_per_crawlmax_requests_per_crawl: NotRequired[int | None]max_session_rotationsmax_session_rotations: NotRequired[int]proxy_configurationproxy_configuration: NotRequired[ProxyConfiguration]request_handlerrequest_handler: NotRequired[Callable[[TCrawlingContext], Awaitable[None]]]request_handler_timeoutrequest_handler_timeout: NotRequired[timedelta]request_providerrequest_provider: NotRequired[RequestProvider]retry_on_blockedretry_on_blocked: NotRequired[bool]session_poolsession_pool: NotRequired[SessionPool]statisticsstatistics: NotRequired[Statistics[StatisticsState]]use_session_pooluse_session_pool: NotRequired[bool]
Copy of the parameter types of
BasicCrawler.__init__meant for typing forwarded init args in subclasses.