Skip to main content

HttpxClient

crawlee.http_clients.httpx_client.HttpxClient

A httpx-based HTTP client used for making HTTP calls in crawlers (BasicCrawler subclasses).

Index

Constructors

Methods

Constructors

__init__

  • __init__(*, persist_cookies_per_session, additional_http_error_status_codes, ignore_http_error_status_codes): None
  • Parameters

    • persist_cookies_per_session: bool = Truekeyword-only
    • additional_http_error_status_codes: Iterable[int] = ()keyword-only
    • ignore_http_error_status_codes: Iterable[int] = ()keyword-only

    Returns None

Methods

crawl

  • async crawl(request, session, proxy_info, statistics): HttpCrawlingResult
  • Parameters

    • request: Request
    • session: Session | None
    • proxy_info: ProxyInfo | None
    • statistics: Statistics

    Returns HttpCrawlingResult

send_request

  • async send_request(url, *, method, headers, session, proxy_info): HttpResponse
  • Parameters

    • url: str
    • method: strkeyword-only
    • headers: httpx.Headers | dict[str, str]keyword-only
    • session: Session | Nonekeyword-only
    • proxy_info: ProxyInfo | Nonekeyword-only

    Returns HttpResponse