Skip to main content
Version: Next

AdaptivePlaywrightCrawlerOptions <ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

Hierarchy

Index

Properties

Properties

optionalinheritedadditionalHttpErrorStatusCodes

additionalHttpErrorStatusCodes?: number[]

An array of additional HTTP response Status Codes to be treated as errors. By default, status codes >= 500 trigger errors.

optionalinheritedblockedStatusCodes

blockedStatusCodes?: number[] = [401, 403, 429]

HTTP status codes that indicate the session should be retired.

A 429 from a domain covered by a ThrottlingRequestManager is handled as a rate limit before this is consulted, so removing 429 here only affects domains that manager does not cover.

optionalinheritedconcurrencySystem

concurrencySystem?: IConcurrencySystem

A pre-configured concurrency governor — the component that decides whether there is free compute for one more task. Typically a ConcurrencySystem, though any IConcurrencySystem is accepted. All scaling configuration (min/max/desired concurrency, scaling ratios, maxTasksPerMinute, snapshotter tuning) lives on the instance itself.

Inject the same instance into several concurrent crawlers to cap their combined concurrency against a single budget. Each crawler still builds and drives its own AutoscaledPool; only the load/scaling accounting is shared.

Mutually exclusive with the minConcurrency/maxConcurrency/initialConcurrency/maxRequestsPerMinute shortcuts, which configure the default system this one replaces — combining the two throws.

You own a supplied system's lifecycle: start() it before run() (which throws otherwise) and stop() it once every crawler borrowing it has finished. The crawler does neither on your behalf.

optionalinheritedconfiguration

configuration?: Configuration

Custom configuration to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedcontextPipelineBuilder

contextPipelineBuilder?: () => ContextPipeline<CrawlingContext<Dictionary>, AdaptivePlaywrightCrawlerContext<any>>

Intended for BasicCrawler subclasses. Prepares a context pipeline that transforms the initial crawling context into the shape given by the Context type parameter.

The option is not required if your crawler subclass does not extend the crawling context with custom information or helpers.

Type declaration

optionalinheritederrorHandler

errorHandler?: ErrorHandler<CrawlingContext<Dictionary>, ExtendedContext>

User-provided function that allows modifying the request object before it gets retried by the crawler. It's executed before each retry for the requests that failed less than maxRequestRetries times.

The function receives the BasicCrawlingContext as the first argument, where the request corresponds to the request to be retried. Second argument is the Error instance that represents the last error thrown during processing of the request.

optionalinheritedeventManager

eventManager?: EventManager

Custom event manager to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedextendContext

extendContext?: (context) => Awaitable<ContextExtension>

Allows the user to extend the crawling context with custom functionality (helpers, references, etc.).

extendContext runs before navigation, so the returned members are visible to the preNavigationHooks, postNavigationHooks, and the requestHandler alike. As a consequence, the context passed to extendContext is the pre-navigation CrawlingContext and does not include navigation-dependent members (e.g. page, response, $, body). If you need those, use a postNavigationHook or the requestHandler instead.

Example usage:

import { BasicCrawler } from 'crawlee';

// Create a crawler instance
const crawler = new BasicCrawler({
    extendContext(context) => ({
        async customHelper() {
            await context.pushData({ url: context.request.url })
        }
    }),
    async requestHandler(context) {
        await context.customHelper();
    },
});

Type declaration

optionalinheritedfailedRequestHandler

failedRequestHandler?: ErrorHandler<CrawlingContext<Dictionary>, ExtendedContext>

A function to handle requests that failed more than maxRequestRetries times.

The function receives the BasicCrawlingContext as the first argument, where the request corresponds to the failed request. Second argument is the Error instance that represents the last error thrown during processing of the request.

optionalinheritedhttpClient

httpClient?: BaseHttpClient

HTTP client implementation for the sendRequest context helper and for plain HTTP crawling. Defaults to ImpitHttpClient when @crawlee/impit-client is installed, otherwise FetchHttpClient.

optionalinheritedid

id?: string

A unique identifier for the crawler instance. This ID is used to isolate the state returned by crawler.useState() from other crawler instances.

When multiple crawler instances use useState() without an explicit id, they will share the same state object for backward compatibility. A warning will be logged in this case.

To ensure each crawler has its own isolated state that also persists across script restarts (e.g., during Apify migrations), provide a stable, unique ID for each crawler instance.

optionalinheritedignoreHttpErrorStatusCodes

ignoreHttpErrorStatusCodes?: number[]

An array of HTTP response Status Codes to be excluded from error consideration. By default, status codes >= 500 trigger errors.

optionalinheritedinitialConcurrency

initialConcurrency?: number

Sets the concurrency (parallelism) the crawl starts with, before any scaling happens. Shortcut for the desiredConcurrency option of the crawler's default ConcurrencySystem. Defaults to minConcurrency.

optionalinheritedkeepAlive

keepAlive?: boolean

Allows to keep the crawler alive even if the RequestQueue gets empty. By default, the crawler.run() will resolve once the queue is empty. With keepAlive: true it will keep running, waiting for more requests to come. Use crawler.stop() to exit the crawler gracefully, or crawler.teardown() to stop it immediately.

optionalinheritedlogger

logger?: CrawleeLogger

Custom logger to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedmaxConcurrency

maxConcurrency?: number

Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the maxConcurrency option of the crawler's default ConcurrencySystem.

optionalinheritedmaxCrawlDepth

maxCrawlDepth?: number

Maximum depth of the crawl. If not set, the crawl will continue until all requests are processed. Setting this to 0 will only process the initial requests, skipping all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests. Passing 1 will process the initial requests and all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests in the handler for initial requests.

optionalinheritedmaxRequestRetries

maxRequestRetries?: number = 3

Specifies the maximum number of retries allowed for a request if its processing fails. This includes retries due to navigation errors, session/proxy errors, or errors thrown from user-supplied functions (requestHandler, preNavigationHooks, postNavigationHooks).

optionalinheritedmaxRequestsPerCrawl

maxRequestsPerCrawl?: number

Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers.

NOTE: In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.

optionalinheritedmaxRequestsPerMinute

maxRequestsPerMinute?: number

The maximum number of requests per minute the crawler should run. By default, this is set to Infinity, but we can pass any positive, non-zero integer. Shortcut for the maxTasksPerMinute option of the crawler's default ConcurrencySystem.

optionalinheritedminConcurrency

minConcurrency?: number

Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the minConcurrency option of the crawler's default ConcurrencySystem.

WARNING: If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically.

optionalinheritedonSkippedRequest

onSkippedRequest?: SkippedRequestCallback

When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped

  1. based on robots.txt file,
  2. because they don't match enqueueLinks filters,
  3. because they are redirected to a URL that doesn't match the enqueueLinks strategy,
  4. or because the maxRequestsPerCrawl limit has been reached

optionalpostNavigationHooks

postNavigationHooks?: AdaptivePostNavigationHook<ContextExtension>[]

Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful. The function accepts a subset of the crawling context. If you attempt to access the page property during HTTP-only crawling, an exception will be thrown. If it's not caught, the request will be transparently retried in a browser.

A hook may optionally return a partial object whose properties are merged into the crawling context (e.g. to override response after solving a challenge).

optionalpreNavigationHooks

preNavigationHooks?: AdaptiveHook<ContextExtension>[]

Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies. The function accepts a subset of the crawling context. If you attempt to access the page property during HTTP-only crawling, an exception will be thrown. If it's not caught, the request will be transparently retried in a browser.

A hook may optionally return a partial object whose properties are merged into the crawling context, allowing the hook to override context members for subsequent hooks and pipeline stages.

optionalinheritedproxyConfiguration

proxyConfiguration?: IProxyConfiguration

If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration.

optionalrenderingTypeDetectionRatio

renderingTypeDetectionRatio?: number

Specifies the frequency of rendering type detection checks - 0.1 means roughly 10% of requests. Defaults to 0.1 (so 10%).

optionalrenderingTypePredictor

renderingTypePredictor?: IRenderingTypePredictor

A custom rendering type predictor. A predictor passed here is borrowed - the crawler never drives its lifecycle, so set it up yourself (the built-in RenderingTypePredictor needs initialize()). Omit the option and the crawler builds its own from renderingTypeDetectionRatio - and initializes it.

optionalinheritedrequestHandler

requestHandler?: RouterHandler<ExtendedContext, Routes> | RequestHandler<ExtendedContext>

User-provided function that performs the logic of the crawler. It is called for each URL to crawl.

The function receives the BasicCrawlingContext as an argument, where the request represents the URL to crawl.

The function must return a promise, which is then awaited by the crawler.

If the function throws an exception, the crawler will try to re-crawl the request later, up to the maxRequestRetries times. If all the retries fail, the crawler calls the function provided to the failedRequestHandler parameter. To make this work, we should always let our function throw exceptions rather than catch them. The exceptions are logged to the request using the Request.pushErrorMessage() function.

optionalinheritedrequestHandlerTimeoutSecs

requestHandlerTimeoutSecs?: number = 60

Timeout in which the function passed as requestHandler needs to finish, in seconds.

optionalinheritedrequestList

requestList?: IRequestLoader

Static list of URLs to be processed.

Deprecated - Use the requestManager option instead. To combine a read-only loader (such as a RequestList) with a writable queue, build a tandem with requestList.toTandem(requestQueue) and pass the result as requestManager. When both requestList and requestQueue are provided, they are combined into a tandem automatically.

optionalinheritedrequestManager

requestManager?: IRequestManager

Manager of requests that should be processed by the crawler. Mutually exclusive with the deprecated requestQueue and requestList options.

If not provided, the crawler will open the default RequestQueue when it is first needed.

optionalinheritedrequestQueue

requestQueue?: RequestQueue

Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.

Deprecated - Use the requestManager option instead. A RequestQueue is itself a request manager, so you can pass it directly as requestManager.

optionalinheritedrespectRobotsTxtFile

respectRobotsTxtFile?: boolean | { userAgent?: string }

If set to true, the crawler will automatically try to fetch the robots.txt file for each domain, and skip those that are not allowed. This also prevents disallowed URLs to be added via enqueueLinks.

If an object is provided, it may contain a userAgent property to specify which user-agent should be used when checking the robots.txt file. If not provided, the default user-agent * will be used.

optionalresultChecker

resultChecker?: (result) => boolean

An optional callback that is called on the storage writes recorded by the request handler in plain HTTP mode (exposed as a read-only StorageTransactionView). If it returns false, the request is retried in a browser. If no callback is specified, every result is considered valid.

Type declaration

optionalresultComparator

resultComparator?: (resultA, resultB) => boolean | equal | different | inconclusive

An optional callback used in rendering type detection. On each detection, the result of the plain HTTP run is compared to that of the browser one. If a callback is provided, the contract is as follows: It the callback returns true or 'equal', the results are considered equal and the target site is considered static. If it returns false or 'different', the target site is considered client-rendered. If it returns 'inconclusive', the detection result won't be used. If no result comparator is specified, but there is a resultChecker, any site where the resultChecker returns true is considered static. If neither resultComparator nor resultChecker are specified, a deep comparison of returned dataset items is used as a default.

For a stricter, ready-made comparator that also takes enqueued requests and key-value store changes into account, see fullResultComparator.

Type declaration

optionalinheritedretryOnBlocked

retryOnBlocked?: boolean

If set to true, the crawler will automatically try to bypass any detected bot protection.

Currently supports:

optionalinheritedsameDomainDelaySecs

sameDomainDelaySecs?: number = 0

Indicates how much time (in seconds) to wait before crawling another same domain request. Subdomains are paced together with the site they belong to.

Offered to the crawler's request manager as a minIntervalEverywhere PacingSignal; a manager that already paces every domain it dispatches to takes it, so no domain ends up with two clocks. Otherwise the crawler wraps its request manager in a ThrottlingRequestManager of its own.

optionalinheritedsessionPool

sessionPool?: ISessionPool

An existing session pool instance to use. When provided, the crawler will use this pool directly instead of creating a new one, enabling session sharing across multiple crawlers. The crawler will not tear down a shared pool — the caller is responsible for its lifecycle.

Accepts the built-in SessionPool or any object implementing the ISessionPool interface, so custom session-management strategies can be plugged in.

optionalshouldPropagateError

shouldPropagateError?: (error, context) => Awaitable<boolean> = (error, context) => Awaitable<boolean>

An optional callback that decides whether an error thrown during the plain HTTP request handler should be propagated (instead of falling back to browser navigation).

If the callback returns true, the error is thrown, triggering the standard retry mechanism. If the callback returns false (or is not provided), the error is logged and the crawler falls back to browser navigation (default behavior).

Type declaration

optionalinheritedstatistics

statistics?: IStatistics<StatisticStateExtension>

A preconfigured statistics instance. When provided, the crawler records into it instead of building its own and will not reset() it between run() calls. Accepts the built-in Statistics or any object implementing IStatistics.

Custom fields declared via stateExtension are carried over to crawler.statistics.state:

const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });

const crawler = new BasicCrawler({
    statistics,
    requestHandler: async () => {
        statistics.state.productsFound++;
    },
});

await crawler.run();
// the custom fields are typed on `crawler.statistics` too
console.log(crawler.statistics.state.productsFound);

optionalinheritedstatusMessageCallback

statusMessageCallback?: StatusMessageCallback<BasicCrawlingContext<Dictionary>, BasicCrawler<BasicCrawlingContext<Dictionary>, Dictionary<never>, BasicCrawlingContext<Dictionary> & Dictionary<never>, Record<string, Dictionary>, {}>>

Allows overriding the default status message. The callback needs to call crawler.setStatusMessage() explicitly. The default status message is provided in the parameters.

const crawler = new CheerioCrawler({
    statusMessageCallback: async (ctx) => {
        return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG'
    },
    statusMessageLoggingInterval: 1, // defaults to 10s
    async requestHandler({ $, enqueueLinks, request, log }) {
        // ...
    },
});

optionalinheritedstatusMessageLoggingInterval

statusMessageLoggingInterval?: number

Defines the length of the interval for calling the setStatusMessage in seconds.

optionalinheritedstorageBackend

storageBackend?: StorageBackend

Custom storage backend to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.

optionalinheritedtaskLoopOptions

taskLoopOptions?: TaskLoopOptions

Lets you override the predicates that steer the crawler's task loop: isTaskReadyFunction (may another request start?) and isFinishedFunction (is the crawl over?). The task itself — fetching a request and running it through the pipeline — is owned by the crawler and cannot be overridden.

Concurrency is configured elsewhere — through the minConcurrency/maxConcurrency/maxRequestsPerMinute shortcuts, or a concurrencySystem for finer control.

optionalinheritedtransactionalStorage

transactionalStorage?: boolean | Partial<StorageWritePolicy> = boolean | Partial<StorageWritePolicy>

Makes the storage writes performed while handling a request atomic with respect to the request succeeding: they are recorded in a StorageTransaction spanning the whole request lifecycle and only applied when the request handler succeeds, so a thrown handler leaves no partial writes behind and a retry does not double-write. Reads within the handler see its own writes.

false disables the mechanism entirely; an object overrides the per-storage-type StorageWritePolicy (e.g. { requestQueue: 'deferred' } for all-or-nothing enqueues). withDirectStorageAccess is the per-call-site escape hatch; useState() is deliberately not transactional.

Page Options