HttpxHttpClient
Hierarchy
- BaseHttpClient
- HttpxHttpClient
Index
Constructors
Methods
Constructors
__init__
A default constructor.
Parameters
keyword-onlypersist_cookies_per_session: bool = True
Whether to persist cookies per HTTP session.
keyword-onlyadditional_http_error_status_codes: Iterable[int] = ()
Additional HTTP status codes to treat as errors.
keyword-onlyignore_http_error_status_codes: Iterable[int] = ()
HTTP status codes to ignore as errors.
keyword-onlyhttp1: bool = True
Whether to enable HTTP/1.1 support.
keyword-onlyhttp2: bool = True
Whether to enable HTTP/2 support.
keyword-onlyheader_generator: HeaderGenerator | None = _DEFAULT_HEADER_GENERATOR
Header generator instance to use for generating common headers.
async_client_kwargs: Any
Additional keyword arguments for
httpx.AsyncClient
.
Returns None
Methods
crawl
Perform the crawling for a given request.
This method is called from
crawler.run()
.Parameters
request: Request
The request to be crawled.
keyword-onlysession: Session | None = None
The session associated with the request.
keyword-onlyproxy_info: ProxyInfo | None = None
The information about the proxy to be used.
keyword-onlystatistics: Statistics | None = None
The statistics object to register status codes.
Returns HttpCrawlingResult
The result of the crawling.
send_request
Send an HTTP request via the client.
This method is called from
context.send_request()
helper.Parameters
url: str
The URL to send the request to.
keyword-onlymethod: HttpMethod = 'GET'
The HTTP method to use.
keyword-onlyheaders: (HttpHeaders | dict[str, str]) | None = None
The headers to include in the request.
keyword-onlypayload: HttpPayload | None = None
The data to be sent as the request body.
keyword-onlysession: Session | None = None
The session associated with the request.
keyword-onlyproxy_info: ProxyInfo | None = None
The information about the proxy to be used.
Returns HttpResponse
The HTTP response received from the server.
HTTP client based on the
HTTPX
library.This client uses the
HTTPX
library to perform HTTP requests in crawlers (BasicCrawler
subclasses) and to manage sessions, proxies, and error handling.See the
BaseHttpClient
class for more common information about HTTP clients.Usage