Skip to main content
Version: Next

BrowserCrawlerOptions <Page, Response, Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

Hierarchy

Index

Properties

Properties

optionalinheritedadditionalHttpErrorStatusCodes

additionalHttpErrorStatusCodes?: number[]

An array of additional HTTP response Status Codes to be treated as errors. By default, status codes >= 500 trigger errors.

optionalinheritedblockedStatusCodes

blockedStatusCodes?: number[] = [401, 403, 429]

HTTP status codes that indicate the session should be retired.

A 429 from a domain covered by a ThrottlingRequestManager is handled as a rate limit before this is consulted, so removing 429 here only affects domains that manager does not cover.

optionalbrowserPool

browserPool?: IBrowserPool<Page>

The browser pool the crawler should serve its pages from. This is the single way to run a pool with non-default options: build one with the factory that matches your crawler (playwrightBrowserPool, puppeteerBrowserPool, stagehandBrowserPool) - it accepts every BrowserPool option and supplies the correct browser plugin itself, so the pool can never mismatch the crawler.

A pool passed in this way is borrowed, not owned: the crawler will not tear it down, which is what makes it shareable across crawlers. Since the crawler then builds nothing itself, the options that configure its own pool (launchContext, headless, remoteBrowser) are rejected rather than silently ignored.

When omitted, the crawler builds - and tears down - a default pool for its own browser.

optionalinheritedconcurrencySystem

concurrencySystem?: IConcurrencySystem

A pre-configured concurrency governor — the component that decides whether there is free compute for one more task. Typically a ConcurrencySystem, though any IConcurrencySystem is accepted. All scaling configuration (min/max/desired concurrency, scaling ratios, maxTasksPerMinute, snapshotter tuning) lives on the instance itself.

Inject the same instance into several concurrent crawlers to cap their combined concurrency against a single budget. Each crawler still builds and drives its own AutoscaledPool; only the load/scaling accounting is shared.

Mutually exclusive with the minConcurrency/maxConcurrency/maxRequestsPerMinute shortcuts, which configure the default system this one replaces — combining the two throws.

You own a supplied system's lifecycle: start() it before run() (which throws otherwise) and stop() it once every crawler borrowing it has finished. The crawler does neither on your behalf.

optionalinheritedconfiguration

configuration?: Configuration

Custom configuration to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedcontextPipelineBuilder

contextPipelineBuilder?: () => ContextPipeline<CrawlingContext<Dictionary>, Context>

Intended for BasicCrawler subclasses. Prepares a context pipeline that transforms the initial crawling context into the shape given by the Context type parameter.

The option is not required if your crawler subclass does not extend the crawling context with custom information or helpers.

Type declaration

optionalerrorHandler

errorHandler?: ErrorHandler<CrawlingContext<Dictionary>, ExtendedContext>

User-provided function that allows modifying the request object before it gets retried by the crawler. It's executed before each retry for the requests that failed less than maxRequestRetries times.

The function receives the BrowserCrawlingContext (actual context will be enhanced with the crawler specific properties) as the first argument, where the request corresponds to the request to be retried. Second argument is the Error instance that represents the last error thrown during processing of the request.

optionalinheritedeventManager

eventManager?: EventManager

Custom event manager to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedextendContext

extendContext?: (context) => Awaitable<ContextExtension>

Allows the user to extend the crawling context with custom functionality (helpers, references, etc.).

extendContext runs before navigation, so the returned members are visible to the preNavigationHooks, postNavigationHooks, and the requestHandler alike. As a consequence, the context passed to extendContext is the pre-navigation CrawlingContext and does not include navigation-dependent members (e.g. page, response, $, body). If you need those, use a postNavigationHook or the requestHandler instead.

Example usage:

import { BasicCrawler } from 'crawlee';

// Create a crawler instance
const crawler = new BasicCrawler({
    extendContext(context) => ({
        async customHelper() {
            await context.pushData({ url: context.request.url })
        }
    }),
    async requestHandler(context) {
        await context.customHelper();
    },
});

Type declaration

optionalfailedRequestHandler

failedRequestHandler?: ErrorHandler<CrawlingContext<Dictionary>, ExtendedContext>

A function to handle requests that failed more than option.maxRequestRetries times.

The function receives the BrowserCrawlingContext (actual context will be enhanced with the crawler specific properties) as the first argument, where the request corresponds to the failed request. Second argument is the Error instance that represents the last error thrown during processing of the request.

optionalinheritedhttpClient

httpClient?: BaseHttpClient

HTTP client implementation for the sendRequest context helper and for plain HTTP crawling. Defaults to ImpitHttpClient when @crawlee/impit-client is installed, otherwise FetchHttpClient.

optionalinheritedid

id?: string

A unique identifier for the crawler instance. This ID is used to isolate the state returned by crawler.useState() from other crawler instances.

When multiple crawler instances use useState() without an explicit id, they will share the same state object for backward compatibility. A warning will be logged in this case.

To ensure each crawler has its own isolated state that also persists across script restarts (e.g., during Apify migrations), provide a stable, unique ID for each crawler instance.

optionalinheritedignoreHttpErrorStatusCodes

ignoreHttpErrorStatusCodes?: number[]

An array of HTTP response Status Codes to be excluded from error consideration. By default, status codes >= 500 trigger errors.

optionalignoreIframes

ignoreIframes?: boolean

Whether to ignore iframes when processing the page content via parseWithCheerio helper. By default, iframes are expanded automatically. Use this option to disable this behavior.

optionalignoreShadowRoots

ignoreShadowRoots?: boolean

Whether to ignore custom elements (and their #shadow-roots) when processing the page content via parseWithCheerio helper. By default, they are expanded automatically. Use this option to disable this behavior.

optionalinheritedkeepAlive

keepAlive?: boolean

Allows to keep the crawler alive even if the RequestQueue gets empty. By default, the crawler.run() will resolve once the queue is empty. With keepAlive: true it will keep running, waiting for more requests to come. Use crawler.stop() to exit the crawler gracefully, or crawler.teardown() to stop it immediately.

optionallaunchContext

launchContext?: BrowserLaunchContext<any, any>

optionalinheritedlogger

logger?: CrawleeLogger

Custom logger to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedmaxConcurrency

maxConcurrency?: number

Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the maxConcurrency option of the crawler's default ConcurrencySystem.

optionalinheritedmaxCrawlDepth

maxCrawlDepth?: number

Maximum depth of the crawl. If not set, the crawl will continue until all requests are processed. Setting this to 0 will only process the initial requests, skipping all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests. Passing 1 will process the initial requests and all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests in the handler for initial requests.

optionalinheritedmaxRequestRetries

maxRequestRetries?: number = 3

Specifies the maximum number of retries allowed for a request if its processing fails. This includes retries due to navigation errors, session/proxy errors, or errors thrown from user-supplied functions (requestHandler, preNavigationHooks, postNavigationHooks).

optionalinheritedmaxRequestsPerCrawl

maxRequestsPerCrawl?: number

Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers.

NOTE: In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.

optionalinheritedmaxRequestsPerMinute

maxRequestsPerMinute?: number

The maximum number of requests per minute the crawler should run. By default, this is set to Infinity, but we can pass any positive, non-zero integer. Shortcut for the maxTasksPerMinute option of the crawler's default ConcurrencySystem.

optionalinheritedminConcurrency

minConcurrency?: number

Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the minConcurrency option of the crawler's default ConcurrencySystem.

WARNING: If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically.

optionalnavigationTimeoutSecs

navigationTimeoutSecs?: number

Timeout for the whole navigation phase, in seconds. A single window shared by the preNavigationHooks, the page navigation, and the postNavigationHooks - so a slow hook eats into the same budget the navigation uses. Separate from the requestHandlerTimeoutSecs, which times only the request handler.

optionalinheritedonSkippedRequest

onSkippedRequest?: SkippedRequestCallback

When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped

  1. based on robots.txt file,
  2. because they don't match enqueueLinks filters,
  3. because they are redirected to a URL that doesn't match the enqueueLinks strategy,
  4. or because the maxRequestsPerCrawl limit has been reached

optionalpostNavigationHooks

postNavigationHooks?: BrowserHook<Context, ContextExtension>[]

Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts crawlingContext as the only parameter.

A hook may optionally return a partial object whose properties are merged into the crawling context. This is useful for overriding context members (e.g. response) after solving a challenge.

Example:

postNavigationHooks: [
    async (crawlingContext) => {
        const { page } = crawlingContext;
        if (hasCaptcha(page)) {
            await solveCaptcha(page);
        }
    },
    async (crawlingContext) => {
        if (await needsRevalidation(crawlingContext)) {
            return { response: await crawlingContext.page.reload() };
        }
    },
]

optionalpreNavigationHooks

preNavigationHooks?: BrowserHook<Context, ContextExtension>[]

Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies or browser properties before navigation. The function receives the crawlingContext; the options object forwarded to page.goto() is available as crawlingContext.gotoOptions and can be mutated in place.

Example:

preNavigationHooks: [
    async ({ page, gotoOptions }) => {
        await page.evaluate((attr) => { window.foo = attr; }, 'bar');
        gotoOptions.timeout = 60_000;
        gotoOptions.waitUntil = 'domcontentloaded';
    },
]

A hook may optionally return a partial object whose properties are merged into the crawling context, allowing the hook to override context members for subsequent hooks and pipeline stages.

The context is built up in the following order: base context (request, session, helpers, ...) -> extendContext -> preNavigationHooks -> navigation -> postNavigationHooks -> requestHandler. This means the members added by extendContext are already available here, but navigation-dependent members (e.g. page, response) are not.

optionalinheritedproxyConfiguration

proxyConfiguration?: IProxyConfiguration

If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration.

optionalremoteBrowser

remoteBrowser?: CrawlerRemoteBrowserOptions

Connect to a remote browser service (Browserbase, Browserless, Steel, …) instead of launching locally.

The crawler builds a RemoteBrowserPool around its own browser plugin, so the connection is always for the right browser — there is no plugin to construct and no way to mismatch the pool with the crawler. Supply the connection details only: a static endpoint URL, a function returning one per launch, or a RemoteBrowserProvider.

Cannot be combined with browserPool. To tune the pool wrapping the remote connection, or to share it across crawlers, build it with the remote factory for your crawler (remotePlaywrightBrowserPool, remotePuppeteerBrowserPool, remoteStagehandBrowserPool) and pass it as browserPool.

optionalrequestHandler

requestHandler?: RouterHandler<ExtendedContext, Routes> | RequestHandler<ExtendedContext>

Function that is called to process each request.

The function receives the BrowserCrawlingContext (actual context will be enhanced with the crawler specific properties) as an argument, where:

  • request is an instance of the Request object with details about the URL to open, HTTP method etc;
  • page is an instance of the Puppeteer Page or Playwright Page;
  • response is an instance of the Puppeteer Response or Playwright Response, which is the main resource response as returned by the respective page.goto() function.

The function must return a promise, which is then awaited by the crawler.

If the function throws an exception, the crawler will try to re-crawl the request later, up to the maxRequestRetries times. If all the retries fail, the crawler calls the function provided to the failedRequestHandler parameter. To make this work, we should always let our function throw exceptions rather than catch them. The exceptions are logged to the request using the Request.pushErrorMessage() function.

optionalinheritedrequestHandlerTimeoutSecs

requestHandlerTimeoutSecs?: number = 60

Timeout in which the function passed as requestHandler needs to finish, in seconds.

optionalinheritedrequestList

requestList?: IRequestLoader

Static list of URLs to be processed.

Deprecated - Use the requestManager option instead. To combine a read-only loader (such as a RequestList) with a writable queue, build a tandem with requestList.toTandem(requestQueue) and pass the result as requestManager. When both requestList and requestQueue are provided, they are combined into a tandem automatically.

optionalinheritedrequestManager

requestManager?: IRequestManager

Manager of requests that should be processed by the crawler. Mutually exclusive with the deprecated requestQueue and requestList options.

If not provided, the crawler will open the default RequestQueue when it is first needed.

optionalinheritedrequestQueue

requestQueue?: RequestQueue

Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.

Deprecated - Use the requestManager option instead. A RequestQueue is itself a request manager, so you can pass it directly as requestManager.

optionalinheritedrespectRobotsTxtFile

respectRobotsTxtFile?: boolean | { userAgent?: string }

If set to true, the crawler will automatically try to fetch the robots.txt file for each domain, and skip those that are not allowed. This also prevents disallowed URLs to be added via enqueueLinks.

If an object is provided, it may contain a userAgent property to specify which user-agent should be used when checking the robots.txt file. If not provided, the default user-agent * will be used.

optionalinheritedretryOnBlocked

retryOnBlocked?: boolean

If set to true, the crawler will automatically try to bypass any detected bot protection.

Currently supports:

optionalinheritedsameDomainDelaySecs

sameDomainDelaySecs?: number = 0

Indicates how much time (in seconds) to wait before crawling another same domain request. Subdomains are paced together with the site they belong to.

Wraps the crawler's request manager in a ThrottlingRequestManager; pass one as requestManager yourself to configure it further.

optionalsaveResponseCookies

saveResponseCookies?: boolean

Defines whether the cookies should be persisted for sessions. Enabled by default.

optionalinheritedsessionPool

sessionPool?: ISessionPool

An existing session pool instance to use. When provided, the crawler will use this pool directly instead of creating a new one, enabling session sharing across multiple crawlers. The crawler will not tear down a shared pool — the caller is responsible for its lifecycle.

Accepts the built-in SessionPool or any object implementing the ISessionPool interface, so custom session-management strategies can be plugged in.

optionalinheritedstatistics

statistics?: IStatistics<StatisticStateExtension>

A preconfigured statistics instance. When provided, the crawler records into it instead of building its own and will not reset() it between run() calls. Accepts the built-in Statistics or any object implementing IStatistics.

Custom fields declared via stateExtension are carried over to crawler.statistics.state:

const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });

const crawler = new BasicCrawler({
    statistics,
    requestHandler: async () => {
        statistics.state.productsFound++;
    },
});

await crawler.run();
// the custom fields are typed on `crawler.statistics` too
console.log(crawler.statistics.state.productsFound);

optionalinheritedstatusMessageCallback

statusMessageCallback?: StatusMessageCallback<BasicCrawlingContext<Dictionary>, BasicCrawler<BasicCrawlingContext<Dictionary>, Dictionary<never>, BasicCrawlingContext<Dictionary> & Dictionary<never>, Record<string, Dictionary>, {}>>

Allows overriding the default status message. The callback needs to call crawler.setStatusMessage() explicitly. The default status message is provided in the parameters.

const crawler = new CheerioCrawler({
    statusMessageCallback: async (ctx) => {
        return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG'
    },
    statusMessageLoggingInterval: 1, // defaults to 10s
    async requestHandler({ $, enqueueLinks, request, log }) {
        // ...
    },
});

optionalinheritedstatusMessageLoggingInterval

statusMessageLoggingInterval?: number

Defines the length of the interval for calling the setStatusMessage in seconds.

optionalinheritedstorageBackend

storageBackend?: StorageBackend

Custom storage backend to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedtaskLoopOptions

taskLoopOptions?: TaskLoopPredicates

Lets you override the predicates that steer the crawler's task loop: isTaskReadyFunction (may another request start?) and isFinishedFunction (is the crawl over?). The task itself — fetching a request and running it through the pipeline — is owned by the crawler and cannot be overridden.

Concurrency is configured elsewhere — through the minConcurrency/maxConcurrency/maxRequestsPerMinute shortcuts, or a concurrencySystem for finer control.

optionalinheritedtransactionalStorage

transactionalStorage?: boolean | Partial<StorageWritePolicy> = boolean | Partial<StorageWritePolicy>

Makes the storage writes performed while handling a request atomic with respect to the request succeeding: they are recorded in a StorageTransaction spanning the whole request lifecycle and only applied when the request handler succeeds, so a thrown handler leaves no partial writes behind and a retry does not double-write. Reads within the handler see its own writes.

false disables the mechanism entirely; an object overrides the per-storage-type StorageWritePolicy (e.g. { requestQueue: 'deferred' } for all-or-nothing enqueues). withDirectStorageAccess is the per-call-site escape hatch; useState() is deliberately not transactional.

Page Options