Sending HTTP requests

Intro

Every crawler passes a sendRequest() helper to the request handler. BasicCrawler doesn't download anything on its own, so this is how it fetches the current request. Other crawlers can use it to send additional requests, for example to call an API.

import { BasicCrawler } from 'crawlee';

const crawler = new BasicCrawler({
    async requestHandler({ sendRequest, log }) {
        const response = await sendRequest();
        log.info('received body', { body: await response.text() });
    },
});

sendRequest() uses the crawler's HTTP client, which is ImpitHttpClient by default (see the HTTP clients guide for the alternatives), together with the cookies, proxy and browser fingerprint of the current session. It returns a standard Response, so you read the body with response.text(), response.json() or response.arrayBuffer(). Like fetch, it follows redirects and doesn't throw on 4xx or 5xx status codes, so check response.ok or response.status yourself.

sendRequest API

sendRequest(
    requestOverrides?: Partial<HttpRequestOptions>,
    optionsOverrides?: SendRequestOptions,
): Promise<Response>

Request overrides

The first argument (HttpRequestOptions) describes the request. Anything you leave out is taken from the request currently being processed.

• url and method replace the URL and the HTTP method.
• headers is a Headers instance merged into the headers of the current request.
• body, json and form replace the payload. Use at most one of them. json and form also set the content-type header, unless you set it yourself.
• searchParams replaces the query string of the URL.
• username and password set the authorization header for HTTP Basic authentication.

import { BasicCrawler } from 'crawlee';

const crawler = new BasicCrawler({
    async requestHandler({ sendRequest, log }) {
        const response = await sendRequest({
            url: 'https://api.example.com/search',
            method: 'POST',
            headers: new Headers({ 'x-api-key': 'my-api-key' }),
            json: { query: 'crawlee' },
        });
        log.info('received results', await response.json());
    },
});

Request options

The second argument (SendRequestOptions) controls how the request is sent.

• proxyUrl sends the request through the given proxy instead of the session's proxy.
• cookieJar replaces the session's cookie jar.
• timeoutMillis aborts the request after the given number of milliseconds, and signal aborts it when the AbortSignal fires.
• ignoreTlsErrors skips TLS certificate checks, if the HTTP client supports it.

The request overrides accept proxyUrl, cookieJar, timeout and signal too. When an option is set in both arguments, the one from the second argument is used.

Proxies

By default, the request goes through the proxy of the current session, which comes from the crawler's proxyConfiguration. To use a different proxy, pass its URL in the format of protocol://username:password@hostname:port:

import { BasicCrawler } from 'crawlee';

const crawler = new BasicCrawler({
    async requestHandler({ sendRequest, log }) {
        const response = await sendRequest({}, { proxyUrl: 'http://auto:password@proxy.apify.com:8000' });
        log.info('received body', { body: await response.text() });
    },
});

Note that a fixed proxyUrl bypasses the proxy rotation done by sessions.

Cookies

By default, cookies are read from and saved to the cookie jar of the current session. You can pass any cookie jar compatible with the tough-cookie CookieJar instead:

import { BasicCrawler } from 'crawlee';
import { CookieJar } from 'tough-cookie';

const cookieJar = new CookieJar();

const crawler = new BasicCrawler({
    async requestHandler({ sendRequest, log }) {
        const response = await sendRequest({}, { cookieJar });
        log.info('received body', { body: await response.text() });
    },
});

Browser fingerprints

The HTTP client is responsible for making requests look like they come from a real browser, so there are no per-request options for that. To change this behavior, configure or replace the crawler's httpClient. See the HTTP clients guide and the avoid getting blocked guide.