Skip to main content
Version: 4.0 (RC)

Avoid getting blocked

A scraper might get blocked for numerous reasons. Let's narrow it down to the two main ones. The first is a bad or blocked IP address. You can learn about this topic in the proxy management guide. The second reason is browser fingerprints (or signatures), which we will explore more in this guide. Check the Apify Academy anti-scraping course to gain a deeper theoretical understanding of blocking and learn a few tips and tricks.

Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.

The two are not handled separately. In Crawlee a Session ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the SessionPool rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the session management guide for how to control the rotation, and the proxy management guide for the IP half.

Using browser fingerprints​

Changing browser fingerprints can be a tedious job. Luckily, Crawlee provides this feature with zero configuration necessary - the usage of fingerprints is enabled by default and available in PlaywrightCrawler and PuppeteerCrawler. So whenever we build a scraper that is using one of these crawlers - the fingerprints are going to be generated for the default browser and the operating system out of the box.

Customizing browser fingerprints​

In certain cases we want to narrow down the fingerprints used - e.g. specify a certain operating system, locale or browser. This is also possible with Crawlee - the crawler can have the generation algorithm customized to reflect the particular browser version and many more. Let's take a look at the examples bellow:

import { PlaywrightCrawler, playwrightBrowserPool } from 'crawlee';
import { BrowserName, DeviceCategory, OperatingSystemsName } from '@crawlee/browser-pool';

const crawler = new PlaywrightCrawler({
    browserPool: playwrightBrowserPool({
        useFingerprints: true, // this is the default
        fingerprintOptions: {
            fingerprintGeneratorOptions: {
                browsers: [
                    {
                        name: BrowserName.edge,
                        minVersion: 96,
                    },
                ],
                devices: [DeviceCategory.desktop],
                operatingSystems: [OperatingSystemsName.windows],
            },
        },
    }),
    // ...
});

Disabling browser fingerprints​

On the contrary, sometimes we want to entirely disable the usage of browser fingerprints. This is easy to do with Crawlee too. All we have to do is build the crawler's browser pool with useFingerprints set to false:

import { PlaywrightCrawler, playwrightBrowserPool } from 'crawlee';

const crawler = new PlaywrightCrawler({
    browserPool: playwrightBrowserPool({
        useFingerprints: false,
    }),
    // ...
});

Fingerprints for HTTP crawlers​

Every session carries a lightweight fingerprint hint — a browser, platform, and device triple — that the request's HTTP client receives and applies on a best-effort basis. By default each session is given a realistic, randomized fingerprint (the host operating system as platform, with a plausible browser/device for it), and it rotates with the session just like the IP and cookies do.

How much of the hint is used depends on the client. The impit HTTP client maps the session's browser hint to a matching TLS and HTTP impersonation profile, so the connection's low-level signature lines up with the headers being sent.

The same hint also drives browser crawlers, where it seeds the generated browser fingerprint. The hint only fixes the broad strokes — the browser family, operating system, and device — so a session presents a coherent profile, but it does not make the two backends produce byte-identical fingerprints: impit and a real browser will still differ in the finer details (a slightly different user-agent string, for example).

You can pin the fingerprint explicitly through sessionOptions when you need a specific profile:

import { CheerioCrawler, SessionPool } from 'crawlee';
import { ImpitHttpClient } from '@crawlee/impit-client';

const crawler = new CheerioCrawler({
    httpClient: new ImpitHttpClient(),
    sessionPool: new SessionPool({
        sessionOptions: {
            fingerprint: { browser: 'firefox', platform: 'windows', device: 'desktop' },
        },
    }),
    requestHandler: async ({ $ }) => {
        // requests impersonate desktop Firefox on Windows
    },
});

Camoufox​

For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try Camoufox, a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our handleCloudflareChallengeHook post-navigation hook, it should be able to successfully mimic the required user action and get you through it. The hook also reloads the page after the challenge clears and propagates the fresh response back into the crawling context.

import { PlaywrightCrawler, handleCloudflareChallengeHook, playwrightBrowserPool } from 'crawlee';
import { launchOptions } from 'camoufox-js';
import { firefox } from 'playwright';

const crawler = new PlaywrightCrawler({
    postNavigationHooks: [handleCloudflareChallengeHook()],
    browserPool: playwrightBrowserPool({
        // Disable the default fingerprint spoofing to avoid conflicts with Camoufox.
        useFingerprints: false,
        launchContext: {
            launcher: firefox,
            launchOptions: await launchOptions({
                headless: true,
            }),
        },
    }),
    // ...
});

Related links