Skip to main content

HttpCrawler

crawlee.http_crawler._http_crawler.HttpCrawler

A crawler that fetches the request URL using httpx.

Index

Constructors

Constructors

__init__

  • __init__(*, additional_http_error_status_codes, ignore_http_error_status_codes, kwargs): None

  • Initialize the HttpCrawler.

    Parameters

    • additional_http_error_status_codes: Iterable[int] = ()keyword-only
    • ignore_http_error_status_codes: Iterable[int] = ()keyword-only
    • kwargs: Unpack[BasicCrawlerOptions[HttpCrawlingContext]]

    Returns None