Skip to main content
Version: Next

AdaptivePlaywrightCrawler <ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

experimental

An extension of PlaywrightCrawler that uses a more limited request handler interface so that it is able to switch to HTTP-only crawling when it detects it may be possible.

Example usage:

const crawler = new AdaptivePlaywrightCrawler({
    renderingTypeDetectionRatio: 0.1,
    async requestHandler({ querySelector, pushData, enqueueLinks, request, log }) {
        // This function is called to extract data from a single web page
        const $prices = await querySelector('span.price')

        await pushData({
            url: request.url,
            price: $prices.filter(':contains("$")').first().text(),
        })

        await enqueueLinks({ selector: '.pagination a' })
    },
});

await crawler.run([
    'http://www.example.com/page-1',
    'http://www.example.com/page-2',
]);

Hierarchy

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

  • new AdaptivePlaywrightCrawler<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>(options): AdaptivePlaywrightCrawler<ContextExtension, ExtendedContext, Routes, StatisticStateExtension>

Properties

optionalreadonlyinheritedproxyConfiguration
experimental

proxyConfiguration?: IProxyConfiguration

A reference to the underlying IProxyConfiguration instance that manages the crawler's proxies. Only available if used by the crawler.

readonlyinheritedrouter
experimental

router: RouterHandler<AdaptivePlaywrightCrawlerContext<any>, Routes> = ...

Default Router instance that will be used if we don't specify any requestHandler. See router.addHandler() and router.addDefaultHandler().

Accessors

inheritedbasicContextPipeline

  • experimental

    The basic part of the context pipeline. Unlike the subclass pipeline, this part has no major side effects (e.g. launching a browser). It also makes typing more explicit, as subclass pipelines expect the basic crawler fields to already be present in the context at runtime.

    Context built with this pipeline can be passed into multiple crawler pipelines at once. This is used e.g. in the AdaptivePlaywrightCrawler.

    Returns ContextPipeline<{ request: CrawleeRequest<Dictionary> }, CrawlingContext<Dictionary>>

inheritedconcurrencySystem

  • experimental

    The concurrency governor the run in progress is booking its requests against — either the concurrencySystem that was injected, or the default the crawler built for itself. Read it for telemetry: desiredConcurrency, currentConcurrency, isRunning.

    NOTE: undefined outside a run, and a crawler-owned default is rebuilt for every run — so read it during a run rather than caching it across runs.

    IConcurrencySystem is deliberately read-only. Tuning concurrency while a crawl is running means owning the instance: build a ConcurrencySystem yourself and inject it, then set minConcurrency/maxConcurrency/desiredConcurrency on your own reference.

    Returns undefined | IConcurrencySystem

inheritedcontextPipeline

inheritedhasFinishedBefore

  • get hasFinishedBefore(): boolean
  • experimental

    Whether a run() on this instance has already finished - a repeated one continues where it left off.

    Returns boolean

inFlightRenderingTypeDetectionCount

  • get inFlightRenderingTypeDetectionCount(): number
  • experimental

    Number of rendering type detections that have not settled yet, including results the predictor is still persisting.

    Returns number

inheritedlog

inheritedrunning

  • get running(): boolean
  • experimental

    Whether a run() is in progress on this instance.

    Returns boolean

inheritedsessionPool

  • experimental

    A reference to the underlying session pool that manages the crawler's sessions. Typed as ISessionPool so custom implementations can be plugged in via the sessionPool constructor option.

    Returns ISessionPool

inheritedstatistics

  • experimental

    The statistics instance collecting the crawler's run statistics - either the injected statistics option or a crawler-built default. Typed as IStatistics so custom implementations can be plugged in.

    Returns IStatistics<StatisticStateExtension>

Methods

inherited[asyncDispose]

  • [asyncDispose](): Promise<void>
  • experimental

    Returns Promise<void>

inheritedaddRequests

  • experimental

    Adds requests to the queue in batches. By default, it will resolve after the initial batch is added, and continue adding the rest in background. You can configure the batch size via batchSize option and the sleep time in between the batches via waitBetweenBatchesMillis. If you want to wait for all batches to be added to the queue, you can use the waitForAllRequestsToBeAdded promise you get in the response object.

    Optionally, the requests can be filtered using include/exclude glob or regexp patterns and an enqueue strategy (both AND-ed together, same as enqueueLinks), relative to baseUrl. Unlike enqueueLinks, there is no implicit "current page" to anchor the strategy to, so strategy defaults to all here.

    This is an alias for calling addRequestsBatched() on the implicit RequestQueue for this crawler instance.

    Parameters

    Returns Promise<CrawlerAddRequestsResult>

destroy

  • destroy(): Promise<void>
  • experimental

    Releases what the crawler owns beyond a single run — in the browser crawlers, the browser pool. A finished run() has already released everything a run owns, so this is only needed before dropping a crawler you could otherwise have run again.

    Returns Promise<void>

drainRenderingDetections

  • drainRenderingDetections(__namedParameters): Promise<void>
  • experimental

    Waits for in-flight rendering type detections to settle, bounded by timeoutMillis (defaults to the internal timeout).

    Parameters

    • __namedParameters: { timeoutMillis?: number } = {}
      • optionaltimeoutMillis: number

    Returns Promise<void>

inheritedexportData

  • exportData<Data>(path, format, options): Promise<Data[]>
  • experimental

    Retrieves all the data from the default crawler Dataset and exports them to the specified format. Supported formats are currently 'json' and 'csv', and will be inferred from the path automatically.

    Parameters

    Returns Promise<Data[]>

inheritedgetData

inheritedgetDataset

inheritedgetRequestManager

inheritedgetRequestQueue

inheritedpause

  • pause(timeoutSecs): Promise<void>
  • experimental

    Stops dispatching new requests, letting the in-progress ones finish. Resolves once they have settled, or rejects after timeoutSecs if they take too long. Unlike stop(), this does not end the run — run() stays pending until resume().

    NOTE: The concurrency system keeps monitoring and autoscaling throughout, since a shared one may still be serving other crawlers.

    Parameters

    • optionaltimeoutSecs: number

    Returns Promise<void>

inheritedpushData

  • pushData(data, datasetIdentifier): Promise<void>

inheritedresume

  • resume(): void
  • experimental

    Resumes a run suspended with pause(), letting the crawler dispatch requests again. A no-op on a crawler that is not paused.

    Returns void

inheritedrun

  • experimental

    Runs the crawler. Returns a promise that resolves once every request has been processed and the crawler's finished-check (taskLoopOptions.isFinishedFunction, or the default "the request manager is empty") reports that the crawl is over.

    We can use the requests parameter to enqueue the initial requests — it is a shortcut for running crawler.addRequests() before crawler.run().

    Calling run() again on the same instance keeps crawling the same request manager - requests the previous run handled (a failed one counts as handled) are not processed again. Purge the queue or open a fresh one if that is what you want.

    Parameters

    Returns Promise<FinalStatistics>

inheritedsetStatusMessage

  • setStatusMessage(message, options): void
  • experimental

    Sets the status message for the current crawler run.

    This method is periodically called by the crawler, every statusMessageLoggingInterval seconds.

    The message is logged and broadcast via the statusMessage event. Integrations such as the Apify SDK subscribe to that event and forward the message to their status-reporting backend (e.g. the Apify platform).

    Parameters

    Returns void

inheritedstop

  • stop(reason): void
  • experimental

    Gracefully stops the current run of the crawler.

    All the tasks active at the time of calling this method will be allowed to finish.

    To stop the crawler immediately, use crawler.teardown() instead.

    Parameters

    • reason: string = 'The crawler has been gracefully stopped.'

    Returns void

teardown

  • teardown(): Promise<void>
  • experimental

    Stops the crawler immediately, but not before rendering type detections already under way (and results the predictor is still persisting) have settled - see drainRenderingDetections(). Requests that are still running are not waited for, unlike stop().

    Returns Promise<void>

inheriteduseState

  • useState<State>(defaultValue): Promise<State>
  • experimental

    Parameters

    • defaultValue: State = ...

    Returns Promise<State>

Page Options