Skip to main content
Version: Next

playwrightUtils

A namespace that contains various utilities for Playwright - the headless Chrome Node API.

Example usage:

import { launchPlaywright, playwrightUtils } from 'crawlee';

// Navigate to https://www.example.com in Playwright with a POST request
const browser = await launchPlaywright();
const page = await browser.newPage();
await playwrightUtils.gotoExtended(page, {
    url: 'https://example.com,
    method: 'POST',
});

Index

Interfaces

Type Aliases

Functions

Interfaces

BlockRequestsOptions

BlockRequestsOptions:

optionalextraUrlPatterns

extraUrlPatterns?: string[]

If you just want to append to the default blocked patterns, use this property.

optionalurlPatterns

urlPatterns?: string[]

The patterns of URLs to block from being loaded by the browser. Only * can be used as a wildcard. It is also automatically added to the beginning and end of the pattern. This limitation is enforced by the DevTools protocol. .png is the same as *.png*.

CompiledScriptParams

CompiledScriptParams:

page

page: Page

request

request: CrawleeRequest<Dictionary>

DirectNavigationOptions

DirectNavigationOptions:

optionalreferer

referer?: string

Referer header value. If provided it will take preference over the referer header value set by page.setExtraHTTPHeaders(headers).

optionaltimeout

timeout?: number

Maximum operation time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. The default value can be changed by using the browserContext.setDefaultNavigationTimeout(timeout), browserContext.setDefaultTimeout(timeout), page.setDefaultNavigationTimeout(timeout) or page.setDefaultTimeout(timeout) methods.

optionalwaitUntil

waitUntil?: domcontentloaded | load | networkidle

When to consider operation succeeded, defaults to load. Events can be either:

  • 'domcontentloaded' - consider operation to be finished when the DOMContentLoaded event is fired.
  • 'load' - consider operation to be finished when the load event is fired.
  • 'networkidle' - consider operation to be finished when there are no network connections for at least 500 ms.

HandleCloudflareChallengeOptions

HandleCloudflareChallengeOptions:

optionalclickCallback

clickCallback?: (page, boundingBox) => Promise<void>

Allows overriding the checkbox clicking. The boundingBox gives you approximate coordinates of the checkbox, use this if you need to adjust the click position.

Type declaration

    • (page, boundingBox): Promise<void>

    • Parameters

      • page: Page
      • boundingBox: { x: number; y: number }
        • x: number
        • y: number

      Returns Promise<void>

optionalclickPositionCallback

clickPositionCallback?: (page) => Promise<null | { x: number; y: number }>

Allows overriding how the checkbox click position is calculated.

Type declaration

    • (page): Promise<null | { x: number; y: number }>

    • Parameters

      • page: Page

      Returns Promise<null | { x: number; y: number }>

optionalisBlockedCallback

isBlockedCallback?: (page) => Promise<boolean>

Allows overriding the detection of Cloudflare "blocked page".

Type declaration

    • (page): Promise<boolean>

    • Parameters

      • page: Page

      Returns Promise<boolean>

optionalisChallengeCallback

isChallengeCallback?: (page) => Promise<boolean>

Allows overriding the detection of Cloudflare "challenge page".

Type declaration

    • (page): Promise<boolean>

    • Parameters

      • page: Page

      Returns Promise<boolean>

optionalpreChallengeSleepSecs

preChallengeSleepSecs?: number

Optional delay (in seconds) before the first click attempt on the challenge checkbox. Defaults to 1s.

optionalsleepSecs

sleepSecs?: number

How long should we wait after the challenge is completed for the final page to load.

optionalverbose

verbose?: boolean

Logging defaults to the debug level, use this flag to log to info level instead.

InfiniteScrollOptions

InfiniteScrollOptions:

optionalbuttonSelector

buttonSelector?: string

Optionally checks and clicks a button if it appears while scrolling. This is required on some websites for the scroll to work.

optionalmaxScrollHeight

maxScrollHeight?: number = 0

How many pixels to scroll down. If 0, will scroll until bottom of page.

optionalscrollDownAndUp

scrollDownAndUp?: boolean = false

If true, it will scroll up a bit after each scroll down. This is required on some websites for the scroll to work.

optionalstopScrollCallback

stopScrollCallback?: () => unknown

This function is called after every scroll and stops the scrolling process if it returns true. The function can be async.

Type declaration

    • (): unknown

    • Returns unknown

optionaltimeoutSecs

timeoutSecs?: number = 0

How many seconds to scroll for. If 0, will scroll until bottom of page.

optionalwaitForSecs

waitForSecs?: number = 4

How many seconds to wait for no new content to load before exit.

InjectFileOptions

InjectFileOptions:

optionalsurviveNavigations

surviveNavigations?: boolean

Enables the injected script to survive page navigations and reloads without need to be re-injected manually. This does not mean, however, that internal state will be preserved. Just that it will be automatically re-injected on each navigation before any other scripts get the chance to execute.

PlaywrightContextUtils

PlaywrightContextUtils:

blockRequests

  • blockRequests(options): Promise<void>

  • Forces the Playwright browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.

    By default, the function will block all URLs including the following patterns:

    [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]

    If you want to extend this list further, use the extraUrlPatterns option, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use the urlPatterns option, which will override the defaults and block only URLs with your custom patterns.

    This function does not use Playwright's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception.

    The function will never block main document loads and their respective redirects.

    Example usage

    preNavigationHooks: [
        async ({ blockRequests }) => {
            // Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
            await blockRequests({
                extraUrlPatterns: ['adsbygoogle.js'],
            });
        },
    ],

    Parameters

    Returns Promise<void>

compileScript

  • Compiles a Playwright script into an async function that may be executed at any time by providing it with the following object:

    {
       page: Page,
       request: Request,
    }

    Where page is a Playwright Page and request is a Request.

    The function is compiled by using the scriptString parameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = the scriptString parameter.

    As a security measure, no globals such as process or require are accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code.

    Custom context may also be provided using the context parameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand.

    Parameters

    Returns CompiledScriptFunction

enqueueLinksByClickingElements

  • The function finds elements matching a specific CSS selector in a Playwright page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided RequestQueue. This is useful to crawl JavaScript heavy pages where links are not available in href elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in href attributes of the page, see enqueueLinks.

    Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.

    IMPORTANT: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page.

    USING HEADFUL BROWSER: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.

    PERFORMANCE: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the * selector, but be prepared to wait minutes to get results on a large and complex page.

    Example usage

    async requestHandler({ enqueueLinksByClickingElements }) {
        await enqueueLinksByClickingElements({
            selector: 'a.product-detail',
            include: [
                'https://www.example.com/handbags/**',
                'https://www.example.com/purses/**',
            ],
        });
    });

    Parameters

    Returns Promise<BatchAddRequestsResult>

    Promise that resolves to BatchAddRequestsResult object.

handleCloudflareChallenge

  • handleCloudflareChallenge(options): Promise<undefined | Response>

  • This helper tries to solve the Cloudflare challenge automatically by clicking on the checkbox. It will try to detect the Cloudflare page, click on the checkbox, and wait for 10 seconds (configurable via sleepSecs option) for the page to load. Use this in the postNavigationHooks, a failures will result in a SessionError which will be automatically retried, so only successful requests will get into the requestHandler.

    On a successfully solved challenge the page is reloaded and the new Response is returned, which can be returned from the hook to update the crawling context's response. For the common case, prefer the pre-wrapped handleCloudflareChallengeHook hook.

    Example usage

    postNavigationHooks: [
        async (context) => ({ response: await context.handleCloudflareChallenge() }),
    ],

    Parameters

    Returns Promise<undefined | Response>

infiniteScroll

  • infiniteScroll(options): Promise<void>

  • Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.

    Parameters

    Returns Promise<void>

injectFile

  • injectFile(filePath, options): Promise<unknown>

  • Injects a JavaScript file into current page. Unlike Playwright's addScriptTag function, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies.

    File contents are cached for up to 10 files to limit file system access.

    Parameters

    Returns Promise<unknown>

injectJQuery

  • injectJQuery(): Promise<unknown>

  • Injects the jQuery library into current page. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors.

    Beware that the injected jQuery object will be set to the window.$ variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.

    The injected jQuery will survive page navigations and reloads.

    Example usage:

    async requestHandler({ page, injectJQuery }) {
        await injectJQuery();
        const title = await page.evaluate(() => {
            return $('head title').text();
        });
    });

    Note that injectJQuery() does not affect the Playwright page.$() function in any way.

    Returns Promise<unknown>

listDownloads

  • listDownloads(): Promise<Download[]>

  • Returns the list of Download objects collected during the current page navigation and request handler.

    Useful for accessing files that the page downloads automatically. For most use cases, prefer re-enqueueing the URL to FileDownload. Use this only when direct access to the Playwright Download object is required.

    Example usage

    requestHandler: async ({ listDownloads }) => {
        for (const download of await listDownloads()) {
            try {
                const stream = await download.createReadStream();
                // stream to storage...
            } catch {
                // download failed or was cancelled
            }
        }
    },

    Returns Promise<Download[]>

parseWithCheerio

  • parseWithCheerio(selector, timeoutMs): Promise<CheerioAPI>

  • Returns Cheerio handle for page.content(), allowing to work with the data same way as with CheerioCrawler. When provided with the selector argument, it waits for it to be available first.

    Example usage:

    async requestHandler({ parseWithCheerio }) {
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • optionalselector: string
    • optionaltimeoutMs: number

    Returns Promise<CheerioAPI>

saveSnapshot

  • saveSnapshot(options): Promise<void>

  • Saves a full screenshot and HTML of the current page into a Key-Value store.

    Parameters

    Returns Promise<void>

waitForSelector

  • waitForSelector(selector, timeoutMs): Promise<void>

  • Wait for an element matching the selector to appear. Timeout defaults to 5s.

    Example usage:

    async requestHandler({ waitForSelector, parseWithCheerio }) {
        await waitForSelector('article h1');
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • selector: string
    • optionaltimeoutMs: number

    Returns Promise<void>

SaveSnapshotOptions

SaveSnapshotOptions:

optionalconfiguration

configuration?: Configuration = Configuration

Configuration of the crawler that will be used to save the snapshot.

optionalkey

key?: string = ‘SNAPSHOT’

Key under which the screenshot and HTML will be saved. .jpg will be appended for screenshot and .html for HTML.

optionalkeyValueStoreName

keyValueStoreName?: null | string = null | string

Name or id of the Key-Value store where snapshot is saved. By default it is saved to default Key-Value store.

optionalsaveHtml

saveHtml?: boolean = true

If true, it will save a full HTML of the current page as a record with key appended by .html.

optionalsaveScreenshot

saveScreenshot?: boolean = true

If true, it will save a full screenshot of the current page as a record with key appended by .jpg.

optionalscreenshotQuality

screenshotQuality?: number = 50

The quality of the image, between 0-100. Higher quality images have bigger size and require more storage.

Type Aliases

CompiledScriptFunction

CompiledScriptFunction: (params) => Promise<unknown>

Type declaration

Functions

blockRequests

  • blockRequests(page, options): Promise<void>

  • This is a Chromium-only feature.

    Using this option with Firefox and WebKit browsers doesn't have any effect. To set up request blocking for these browsers, use page.route() instead.

    Forces the Playwright browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.

    By default, the function will block all URLs including the following patterns:

    [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]

    If you want to extend this list further, use the extraUrlPatterns option, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use the urlPatterns option, which will override the defaults and block only URLs with your custom patterns.

    This function does not use Playwright's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception.

    The function will never block main document loads and their respective redirects.

    Example usage

    import { launchPlaywright, playwrightUtils } from 'crawlee';

    const browser = await launchPlaywright();
    const page = await browser.newPage();

    // Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
    await playwrightUtils.blockRequests(page, {
        extraUrlPatterns: ['adsbygoogle.js'],
    });

    await page.goto('https://cnn.com');

    Parameters

    Returns Promise<void>

compileScript

  • Compiles a Playwright script into an async function that may be executed at any time by providing it with the following object:

    {
       page: Page,
       request: Request,
    }

    Where page is a Playwright Page and request is a Request.

    The function is compiled by using the scriptString parameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = the scriptString parameter.

    As a security measure, no globals such as process or require are accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code.

    Custom context may also be provided using the context parameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand.

    Parameters

    Returns CompiledScriptFunction

enqueueLinksByClickingElements

  • The function finds elements matching a specific CSS selector in a Playwright page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided RequestQueue. This is useful to crawl JavaScript heavy pages where links are not available in href elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in href attributes of the page, see enqueueLinks.

    Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.

    IMPORTANT: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page.

    USING HEADFUL BROWSER: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.

    PERFORMANCE: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the * selector, but be prepared to wait minutes to get results on a large and complex page.

    Example usage

    await playwrightUtils.enqueueLinksByClickingElements({
      page,
      requestManager,
      selector: 'a.product-detail',
      include: [
          'https://www.example.com/handbags/*',
          'https://www.example.com/purses/*',
      ],
    });

    Parameters

    Returns Promise<BatchAddRequestsResult>

    Promise that resolves to BatchAddRequestsResult object.

gotoExtended

  • gotoExtended(page, request, gotoOptions): Promise<Response | null>

  • Extended version of Playwright's page.goto() allowing to perform requests with HTTP method other than GET, with custom headers and POST payload. URL, method, headers and payload are taken from request parameter that must be an instance of Request class.

    NOTE: In recent versions of Playwright using requests other than GET, overriding headers and adding payloads disables browser cache which degrades performance.

    Parameters

    Returns Promise<Response | null>

infiniteScroll

  • infiniteScroll(page, options): Promise<void>

  • Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.

    Parameters

    Returns Promise<void>

injectFile

  • injectFile(page, filePath, options): Promise<unknown>

  • Injects a JavaScript file into a Playwright page. Unlike Playwright's addScriptTag function, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies.

    File contents are cached for up to 10 files to limit file system access.

    Parameters

    Returns Promise<unknown>

injectJQuery

  • injectJQuery(page, options): Promise<unknown>

  • Injects the jQuery library into a Playwright page. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors.

    Beware that the injected jQuery object will be set to the window.$ variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.

    The injected jQuery will survive page navigations and reloads by default.

    Example usage:

    await playwrightUtils.injectJQuery(page);
    const title = await page.evaluate(() => {
      return $('head title').text();
    });

    Note that injectJQuery() does not affect the Playwright page.$() function in any way.

    Parameters

    • page: Page

      Playwright Page object.

    • optionaloptions: { surviveNavigations?: boolean }
      • optionalsurviveNavigations: boolean

        Opt-out option to disable the JQuery reinjection after navigation.

    Returns Promise<unknown>

parseWithCheerio

  • parseWithCheerio(page, ignoreShadowRoots, ignoreIframes): Promise<CheerioAPI>

  • Returns Cheerio handle for page.content(), allowing to work with the data same way as with CheerioCrawler.

    Example usage:

    const $ = await playwrightUtils.parseWithCheerio(page);
    const title = $('title').text();

    Parameters

    • page: Page

      Playwright Page object.

    • ignoreShadowRoots: boolean = false
    • ignoreIframes: boolean = false

    Returns Promise<CheerioAPI>

saveSnapshot

  • saveSnapshot(page, options): Promise<void>

  • Saves a full screenshot and HTML of the current page into a Key-Value store.

    Parameters

    Returns Promise<void>

Page Options