Skip to main content
Version: Next

PlaywrightCrawler <ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

Provides a simple framework for parallel crawling of web pages using headless Chromium, Firefox and Webkit browsers with Playwright. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.

Since Playwright uses headless browser to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using CheerioCrawler, which downloads the pages using raw HTTP requests and is about 10x faster.

The source URLs are represented using Request objects that are fed from the request manager provided via the requestManager constructor option (a RequestQueue is itself a request manager). To read from a read-only source such as a RequestList while still being able to enqueue new requests, combine it with a queue into a RequestManagerTandem via requestLoader.toTandem() and pass the result as requestManager.

The requestList and requestQueue options are deprecated; they are still accepted and folded into a single requestManager for back-compat.

The crawler finishes when there are no more Request objects to crawl.

PlaywrightCrawler opens a new Chrome page (i.e. tab) for each Request object to crawl and then calls the function provided by user as the PlaywrightCrawlerOptions.requestHandler option.

New pages are only opened when there is enough free CPU and memory available, as judged by the crawler's ConcurrencySystem. Concurrency is tuned via the minConcurrency, maxConcurrency and maxRequestsPerMinute options of the PlaywrightCrawler constructor, or, for finer control, by injecting a pre-configured concurrencySystem.

Note that the pool of Playwright instances is internally managed by the BrowserPool class.

Example usage:

const crawler = new PlaywrightCrawler({
    async requestHandler({ page, request }) {
        // This function is called to extract data from a single web page
        // 'page' is an instance of Playwright.Page with page.goto(request.url) already called
        // 'request' is an instance of Request class with information about the page to load
        await Dataset.pushData({
            title: await page.title(),
            url: request.url,
            succeeded: true,
        })
    },
    async failedRequestHandler({ request }) {
        // This function is called when the crawling of a request failed too many times
        await Dataset.pushData({
            url: request.url,
            succeeded: false,
            errors: request.errorMessages,
        })
    },
});

await crawler.run([
    'http://www.example.com/page-1',
    'http://www.example.com/page-2',
]);

Hierarchy

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

  • new PlaywrightCrawler<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>(options): PlaywrightCrawler<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

  • All PlaywrightCrawler parameters are passed via an options object.

    Parameters

    Returns PlaywrightCrawler<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

Properties

inheritedhasFinishedBefore

hasFinishedBefore: boolean = false

inheritedlaunchContext

launchContext: BrowserLaunchContext<LaunchOptions, unknown>

optionalreadonlyinheritedproxyConfiguration

proxyConfiguration?: IProxyConfiguration

A reference to the underlying IProxyConfiguration instance that manages the crawler's proxies. Only available if used by the crawler.

readonlyinheritedrouter

router: RouterHandler<PlaywrightCrawlingContext<any>, Routes> = ...

Default Router instance that will be used if we don't specify any requestHandler. See router.addHandler() and router.addDefaultHandler().

inheritedrunning

running: boolean = false

Accessors

inheritedbasicContextPipeline

  • The basic part of the context pipeline. Unlike the subclass pipeline, this part has no major side effects (e.g. launching a browser). It also makes typing more explicit, as subclass pipelines expect the basic crawler fields to already be present in the context at runtime.

    Context built with this pipeline can be passed into multiple crawler pipelines at once. This is used e.g. in the AdaptivePlaywrightCrawler.

    Returns ContextPipeline<{ request: CrawleeRequest<Dictionary> }, CrawlingContext<Dictionary>>

inheritedbrowserPool

  • A reference to the underlying browser pool that manages the crawler's browsers. Typed as IBrowserPool so custom implementations can be plugged in via the browserPool constructor option.

    Returns IBrowserPool<Page>

inheritedconcurrencySystem

  • The concurrency governor this run is booking its requests against — either the concurrencySystem that was injected, or the default the crawler built for itself. Read it for telemetry: desiredConcurrency, currentConcurrency, isRunning.

    NOTE: undefined until crawler.run() has resolved it. A crawler-owned default is also rebuilt for every run, so the instance is not stable across runs.

    IConcurrencySystem is deliberately read-only. Tuning concurrency while a crawl is running means owning the instance: build a ConcurrencySystem yourself and inject it, then set minConcurrency/maxConcurrency/desiredConcurrency on your own reference.

    Returns undefined | IConcurrencySystem

inheritedcontextPipeline

inheritedlog

inheritedsessionPool

  • A reference to the underlying session pool that manages the crawler's sessions. Typed as ISessionPool so custom implementations can be plugged in via the sessionPool constructor option.

    Returns ISessionPool

inheritedstatistics

  • The statistics instance collecting the crawler's run statistics - either the injected statistics option or a crawler-built default. Typed as IStatistics so custom implementations can be plugged in.

    Returns IStatistics<StatisticStateExtension>

Methods

inheritedaddRequests

  • Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in background. You can configure the batch size via batchSize option and the sleep time in between the batches via waitBetweenBatchesMillis. If you want to wait for all batches to be added to the queue, you can use the waitForAllRequestsToBeAdded promise you get in the response object.

    Optionally, the requests can be filtered using include/exclude glob or regexp patterns and an enqueue strategy (both AND-ed together, same as enqueueLinks), relative to baseUrl. Unlike enqueueLinks, there is no implicit "current page" to anchor the strategy to, so strategy defaults to all here.

    This is an alias for calling addRequestsBatched() on the implicit RequestQueue for this crawler instance.

    Parameters

    • requests: ReadonlyDeep<TypedRequestsLike<Routes>>

      The requests to add

    • options: CrawlerAddRequestsOptions = {}

      Options for the request queue

    Returns Promise<CrawlerAddRequestsResult>

inheritedexportData

  • exportData<Data>(path, format, options): Promise<Data[]>

  • Retrieves all the data from the default crawler Dataset and exports them to the specified format. Supported formats are currently 'json' and 'csv', and will be inferred from the path automatically.

    Parameters

    Returns Promise<Data[]>

inheritedgetData

inheritedgetDataset

inheritedgetRequestManager

inheritedgetRequestQueue

inheritedpause

  • pause(timeoutSecs): Promise<void>

  • Stops dispatching new requests, letting the in-progress ones finish. Resolves once they have settled, or rejects after timeoutSecs if they take too long. Unlike stop(), this does not end the run — run() stays pending until resume().

    NOTE: The concurrency system keeps monitoring and autoscaling throughout, since a shared one may still be serving other crawlers.

    Parameters

    • optionaltimeoutSecs: number

    Returns Promise<void>

inheritedpushData

  • pushData(data, datasetIdentifier): Promise<void>

inheritedresume

  • resume(): void

  • Resumes a run suspended with pause(), letting the crawler dispatch requests again. A no-op on a crawler that is not paused.

    Returns void

inheritedrun

  • Runs the crawler. Returns a promise that resolves once every request has been processed and the crawler's finished-check (taskLoopOptions.isFinishedFunction, or the default "the request manager is empty") reports that the crawl is over.

    We can use the requests parameter to enqueue the initial requests — it is a shortcut for running crawler.addRequests() before crawler.run().

    Parameters

    • optionalrequests: TypedRequestsLike<Routes>

      The requests to add.

    • optionaloptions: CrawlerRunOptions

      Options for the request queue.

    Returns Promise<FinalStatistics>

inheritedsetStatusMessage

  • setStatusMessage(message, options): void

  • Sets the status message for the current crawler run.

    This method is periodically called by the crawler, every statusMessageLoggingInterval seconds.

    The message is logged and broadcast via the statusMessage event. Integrations such as the Apify SDK subscribe to that event and forward the message to their status-reporting backend (e.g. the Apify platform).

    Parameters

    Returns void

inheritedstop

  • stop(reason): void

  • Gracefully stops the current run of the crawler.

    All the tasks active at the time of calling this method will be allowed to finish.

    To stop the crawler immediately, use crawler.teardown() instead.

    Parameters

    • reason: string = 'The crawler has been gracefully stopped.'

    Returns void

inheriteduseState

  • useState<State>(defaultValue): Promise<State>

  • Parameters

    • defaultValue: State = ...

    Returns Promise<State>

Page Options