Skip to main content
Version: Next

StagehandCrawlerOptions

Options for StagehandCrawler.

Hierarchy

Index

Properties

Properties

optionalinheritedautoscaledPoolOptions

autoscaledPoolOptions?: AutoscaledPoolOptions

Custom options passed to the underlying AutoscaledPool constructor.

NOTE: The runTaskFunction option is provided by the crawler and cannot be overridden. However, we can provide custom implementations of isFinishedFunction and isTaskReadyFunction.

optionalinheritedbrowserPoolOptions

browserPoolOptions?: Partial<BrowserPoolOptions<BrowserPlugin<CommonLibrary, undefined | Dictionary, CommonBrowser, unknown, CommonPage>>> & Partial<BrowserPoolHooks<never, never, never>>

Custom options passed to the underlying BrowserPool constructor. We can tweak those to fine-tune browser management.

optionalinheritederrorHandler

errorHandler?: BrowserErrorHandler<StagehandCrawlingContext<Dictionary>>

User-provided function that allows modifying the request object before it gets retried by the crawler. It's executed before each retry for the requests that failed less than maxRequestRetries times.

The function receives the BrowserCrawlingContext (actual context will be enhanced with the crawler specific properties) as the first argument, where the request corresponds to the request to be retried. Second argument is the Error instance that represents the last error thrown during processing of the request.

optionalinheritedexperiments

experiments?: CrawlerExperiments

Enables experimental features of Crawlee, which can alter the behavior of the crawler. WARNING: these options are not guaranteed to be stable and may change or be removed at any time.

optionalfailedRequestHandler

failedRequestHandler?: StagehandRequestHandler

Function called when request handling fails after all retries.

optionalinheritedheadless

headless?: boolean | new | old

Whether to run browser in headless mode. Defaults to true. Can be also set via Configuration.

optionalinheritedhttpClient

httpClient?: BaseHttpClient

HTTP client implementation for the sendRequest context helper and for plain HTTP crawling. Defaults to a new instance of GotScrapingHttpClient

optionalinheritedignoreIframes

ignoreIframes?: boolean

Whether to ignore iframes when processing the page content via parseWithCheerio helper. By default, iframes are expanded automatically. Use this option to disable this behavior.

optionalinheritedignoreShadowRoots

ignoreShadowRoots?: boolean

Whether to ignore custom elements (and their #shadow-roots) when processing the page content via parseWithCheerio helper. By default, they are expanded automatically. Use this option to disable this behavior.

optionalinheritedkeepAlive

keepAlive?: boolean

Allows to keep the crawler alive even if the RequestQueue gets empty. By default, the crawler.run() will resolve once the queue is empty. With keepAlive: true it will keep running, waiting for more requests to come. Use crawler.stop() to exit the crawler gracefully, or crawler.teardown() to stop it immediately.

optionallaunchContext

launchContext?: StagehandLaunchContext

Launch context with Stagehand-specific options.

optionalinheritedmaxConcurrency

maxConcurrency?: number

Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool maxConcurrency option.

optionalinheritedmaxCrawlDepth

maxCrawlDepth?: number

Maximum depth of the crawl. If not set, the crawl will continue until all requests are processed. Setting this to 0 will only process the initial requests, skipping all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests. Passing 1 will process the initial requests and all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests in the handler for initial requests.

optionalinheritedmaxRequestRetries

maxRequestRetries?: number = 3

Specifies the maximum number of retries allowed for a request if its processing fails. This includes retries due to navigation errors or errors thrown from user-supplied functions (requestHandler, preNavigationHooks, postNavigationHooks).

This limit does not apply to retries triggered by session rotation (see maxSessionRotations).

optionalinheritedmaxRequestsPerCrawl

maxRequestsPerCrawl?: number

Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers.

NOTE: In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.

optionalinheritedmaxRequestsPerMinute

maxRequestsPerMinute?: number

The maximum number of requests per minute the crawler should run. By default, this is set to Infinity, but we can pass any positive, non-zero integer. Shortcut for the AutoscaledPool maxTasksPerMinute option.

optionalinheritedmaxSessionRotations

maxSessionRotations?: number = 10

Maximum number of session rotations per request. The crawler will automatically rotate the session in case of a proxy error or if it gets blocked by the website.

The session rotations are not counted towards the maxRequestRetries limit.

optionalinheritedminConcurrency

minConcurrency?: number

Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the AutoscaledPool minConcurrency option.

WARNING: If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically.

optionalinheritednavigationTimeoutSecs

navigationTimeoutSecs?: number

Timeout in which page navigation needs to finish, in seconds.

optionalinheritedonSkippedRequest

onSkippedRequest?: SkippedRequestCallback

When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped

  1. based on robots.txt file,
  2. because they don't match enqueueLinks filters,
  3. because they are redirected to a URL that doesn't match the enqueueLinks strategy,
  4. or because the maxRequestsPerCrawl limit has been reached

optionalinheritedpersistCookiesPerSession

persistCookiesPerSession?: boolean

Defines whether the cookies should be persisted for sessions. This can only be used when useSessionPool is set to true.

optionalpostNavigationHooks

postNavigationHooks?: StagehandHook[]

Async functions that are sequentially evaluated after the navigation.

optionalpreNavigationHooks

preNavigationHooks?: StagehandHook[]

Async functions that are sequentially evaluated before the navigation.

optionalinheritedproxyConfiguration

proxyConfiguration?: ProxyConfiguration

If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration.

optionalrequestHandler

requestHandler?: StagehandRequestHandler

Function that is called to process each request.

The function receives the StagehandCrawlingContext as an argument, where:

  • request is an instance of the Request object with details about the URL to open, HTTP method etc.
  • page is an enhanced Playwright Page with AI methods
  • browserController is an instance of StagehandController
  • response is the main resource response as returned by page.goto(request.url)
  • stagehand is the Stagehand instance for advanced control

The page object is enhanced with AI-powered methods:

  • page.act(instruction) - Perform actions using natural language
  • page.extract(instruction, schema) - Extract structured data
  • page.observe() - Get AI-suggested actions
  • page.agent(config) - Create autonomous agents

The function must return a promise, which is then awaited by the crawler.

If the function throws an exception, the crawler will try to re-crawl the request later, up to option.maxRequestRetries times.

@example
async requestHandler({ request, page, log }) {
  log.info(`Processing ${request.url}`);

  // Use AI-powered actions
  await page.act('Click the Products menu');

  // Extract structured data
  const products = await page.extract(
    'Get all products',
    z.object({
      items: z.array(z.object({
        name: z.string(),
        price: z.number(),
      })),
    })
  );

  // Mix with standard Playwright methods
  await page.screenshot({ path: 'products.png' });
}

optionalinheritedrequestHandlerTimeoutSecs

requestHandlerTimeoutSecs?: number = 60

Timeout in which the function passed as requestHandler needs to finish, in seconds.

optionalinheritedrequestList

requestList?: IRequestList

Static list of URLs to be processed. If not provided, the crawler will open the default request queue when the crawler.addRequests() function is called.

Alternatively, requests parameter of crawler.run() could be used to enqueue the initial requests - it is a shortcut for running crawler.addRequests() before the crawler.run().

optionalinheritedrequestManager

requestManager?: IRequestManager

Allows explicitly configuring a request manager. Mutually exclusive with the requestQueue and requestList options.

This enables explicitly configuring the crawler to use RequestManagerTandem, for instance. If using this, the type of BasicCrawler.requestQueue may not be fully compatible with the RequestProvider class.

optionalinheritedrequestQueue

requestQueue?: RequestProvider

Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. If not provided, the crawler will open the default request queue when the crawler.addRequests() function is called.

Alternatively, requests parameter of crawler.run() could be used to enqueue the initial requests - it is a shortcut for running crawler.addRequests() before the crawler.run().

optionalinheritedrespectRobotsTxtFile

respectRobotsTxtFile?: boolean | { userAgent?: string }

If set to true, the crawler will automatically try to fetch the robots.txt file for each domain, and skip those that are not allowed. This also prevents disallowed URLs to be added via enqueueLinks.

If an object is provided, it may contain a userAgent property to specify which user-agent should be used when checking the robots.txt file. If not provided, the default user-agent * will be used.

optionalinheritedretryOnBlocked

retryOnBlocked?: boolean

If set to true, the crawler will automatically try to bypass any detected bot protection.

Currently supports:

optionalinheritedsameDomainDelaySecs

sameDomainDelaySecs?: number = 0

Indicates how much time (in seconds) to wait before crawling another same domain request.

optionalinheritedsessionPoolOptions

sessionPoolOptions?: SessionPoolOptions

The configuration options for SessionPool to use.

optionalstagehandOptions

stagehandOptions?: StagehandOptions

Stagehand-specific configuration options. These options configure the AI behavior and Browserbase integration.

optionalinheritedstatisticsOptions

statisticsOptions?: StatisticsOptions

Customize the way statistics collecting works, such as logging interval or whether to output them to the Key-Value store.

optionalinheritedstatusMessageCallback

statusMessageCallback?: StatusMessageCallback<BasicCrawlingContext<Dictionary>, BasicCrawler<BasicCrawlingContext<Dictionary>>>

Allows overriding the default status message. The callback needs to call crawler.setStatusMessage() explicitly. The default status message is provided in the parameters.

const crawler = new CheerioCrawler({
    statusMessageCallback: async (ctx) => {
        return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG'
    },
    statusMessageLoggingInterval: 1, // defaults to 10s
    async requestHandler({ $, enqueueLinks, request, log }) {
        // ...
    },
});

optionalinheritedstatusMessageLoggingInterval

statusMessageLoggingInterval?: number

Defines the length of the interval for calling the setStatusMessage in seconds.

optionalinheriteduseSessionPool

useSessionPool?: boolean

Basic crawler will initialize the SessionPool with the corresponding sessionPoolOptions. The session instance will be than available in the requestHandler.

Page Options