Using a custom HTTP client (Experimental)

The BasicCrawler class allows you to configure the HTTP client implementation using the httpClient constructor option. This might be useful for testing or if you need to swap out the default implementation based on got-scraping for something else, such as curl-impersonate.

Built-in HTTP clients

Crawlee provides several HTTP client implementations out of the box:

• ImpitHttpClient (default) - Uses the impit library for making requests that closely mimic browser behavior.
• GotScrapingHttpClient - Uses the got-scraping library for browser-like requests with support for custom headers, browser fingerprints, and proxies. This was the default HTTP client in Crawlee v3.
• FetchHttpClient - Simple implementation using the native fetch API (does not support proxies).

Implementing a custom HTTP client

To create a custom HTTP client, extend the BaseHttpClient abstract class from @crawlee/http-client. The base class handles common functionality like cookie management, redirect following, session integration, proxy support, and timeout handling.

Your custom implementation only needs to override the fetch method to perform the actual network request:

import { BaseHttpClient, type CustomFetchOptions } from '@crawlee/http-client';

/**
 * A simple HTTP client implementation using the native `fetch` API.
 *
 * Custom implementations only need to override the `fetch` method.
 */
export class CustomFetchClient extends BaseHttpClient {
    protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise<Response> {
        // The base class handles cookies, redirects, sessions, and timeouts.
        // We only need to perform the actual network request here.
        return fetch(request, options);
    }
}

By extending BaseHttpClient, your implementation automatically gets:

• Cookie jar management (applying cookies before requests, saving cookies from responses)
• Automatic redirect following (up to 10 redirects)
• Session integration (proxy URL and cookies from session)
• Timeout handling via AbortSignal
• Proxy URL support

You may then instantiate it and pass to a crawler constructor:

import { HttpCrawler } from 'crawlee';
import { CustomFetchClient } from './implementation.js';

const crawler = new HttpCrawler({
    httpClient: new CustomFetchClient(),
    async requestHandler() {
        /* ... */
    },
});

Alternatively, you can implement the BaseHttpClient interface directly if you need full control over all aspects of the HTTP request handling, including cookies, redirects, and sessions. However, this approach requires implementing significantly more logic yourself.