Skip to main content
Version: Next

PuppeteerCrawlingContext <UserData>

Hierarchy

Index

Properties

Methods

Properties

inheritedaddRequests

addRequests: (requestsLike, options) => Promise<AddRequestsBatchedResult>

Add requests directly to the request queue currently used by the crawler.

Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way enqueueLinks does for extracted links.

Type declaration

inheritedenqueueLinks

enqueueLinks: (options) => Promise<AddRequestsBatchedResult>

Helper function for extracting URLs from the current page and adding them to the request queue.

Type declaration

inheritedextractLinks

extractLinks: (options) => Promise<string[]>

Extracts URLs from the current page, without adding them to the request queue.

Type declaration

    • (options): Promise<string[]>

inheritedgetKeyValueStore

getKeyValueStore: (identifier) => Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

Get a key-value store with given name or id, or the default one for the crawler.

Type declaration

    • (identifier): Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

    • Parameters

      Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

inheritedgotoOptions

gotoOptions: GoToOptions

Options object passed to the underlying page.goto() call. preNavigationHooks can mutate this object (or return { gotoOptions: ... }) to influence the navigation.

inheritedid

id: string

inheritedlog

log: CrawleeLogger

A preconfigured logger for the request handler.

inheritedpage

page: Page

The browser page object where the web page is loaded and rendered.

optionalinheritedproxyInfo

proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.

inheritedrequest

request: LoadedRequest<CrawleeRequest<UserData>>

The request object that was successfully loaded and navigated to, including the loadedUrl property.

inheritedresponse

response: HTTPResponse

The HTTP response object returned by the browser's navigation.

inheritedsendRequest

sendRequest: (requestOverrides, optionsOverrides) => Promise<Response>

Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.

This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.

async requestHandler({ sendRequest }) {
    const { body } = await sendRequest({
        // override headers only
        headers: { ... },
    });
},

Type declaration

    • (requestOverrides, optionsOverrides): Promise<Response>

inheritedsession

session: ISession

inheriteduseState

useState: <State>(defaultValue) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.

Type declaration

    • <State>(defaultValue): Promise<State>

    • Parameters

      • optionaldefaultValue: State

      Returns Promise<State>

Methods

inheritedaddInterceptRequestHandler

  • addInterceptRequestHandler(handler): Promise<void>

  • Adds request interception handler in similar to page.on('request', handler); but in addition to that supports multiple parallel handlers.

    All the handlers are executed sequentially in the order as they were added. Each of the handlers must call one of request.continue(), request.abort() and request.respond(). In addition to that any of the handlers may modify the request object (method, postData, headers) by passing its overrides to request.continue(). If multiple handlers modify same property then the last one wins. Headers are merged separately so you can override only a value of specific header.

    If one the handlers calls request.abort() or request.respond() then request is not propagated further to any of the remaining handlers.

    Example usage:

    preNavigationHooks: [
        async ({ addInterceptRequestHandler }) => {
            // Replace images with placeholder.
            await addInterceptRequestHandler((request) => {
                if (request.resourceType() === 'image') {
                    return request.respond({
                        statusCode: 200,
                        contentType: 'image/jpeg',
                        body: placeholderImageBuffer,
                    });
                }
                return request.continue();
            });

            // Abort all the scripts.
            await addInterceptRequestHandler((request) => {
                if (request.resourceType() === 'script') return request.abort();
                return request.continue();
            });

            // Change requests to post.
            await addInterceptRequestHandler((request) => {
                return request.continue({
                     method: 'POST',
                });
            });
        }),
    ],

    Parameters

    Returns Promise<void>

inheritedblockRequests

  • blockRequests(options): Promise<void>

  • Forces the Puppeteer browser tab to block loading URLs that match a provided pattern. This is useful to speed up crawling of websites, since it reduces the amount of data that needs to be downloaded from the web, but it may break some websites or unexpectedly prevent loading of resources.

    By default, the function will block all URLs including the following patterns:

    [".css", ".jpg", ".jpeg", ".png", ".svg", ".gif", ".woff", ".pdf", ".zip"]

    If you want to extend this list further, use the extraUrlPatterns option, which will keep blocking the default patterns, as well as add your custom ones. If you would like to block only specific patterns, use the urlPatterns option, which will override the defaults and block only URLs with your custom patterns.

    This function does not use Puppeteer's request interception and therefore does not interfere with browser cache. It's also faster than blocking requests using interception, because the blocking happens directly in the browser without the round-trip to Node.js, but it does not provide the extra benefits of request interception.

    The function will never block main document loads and their respective redirects.

    Example usage

    preNavigationHooks: [
        async ({ blockRequests }) => {
            // Block all requests to URLs that include `adsbygoogle.js` and also all defaults.
            await blockRequests({
                extraUrlPatterns: ['adsbygoogle.js'],
            }),
        }),
    ],

    Parameters

    Returns Promise<void>

inheritedcloseCookieModals

  • closeCookieModals(): Promise<void>

  • Tries to close cookie consent modals on the page. Based on the I Don't Care About Cookies browser extension.

    Note that this method requires the idcac-playwright package to be installed. Crawlee does not include it by default due to licensing issues.

    To use this method, please install the package manually by running:

    npm install idcac-playwright

    Returns Promise<void>

inheritedcompileScript

  • Compiles a Puppeteer script into an async function that may be executed at any time by providing it with the following object:

    {
       page: Page,
       request: Request,
    }

    Where page is a Puppeteer Page and request is a Request.

    The function is compiled by using the scriptString parameter as the function's body, so any limitations to function bodies apply. Return value of the compiled function is the return value of the function body = the scriptString parameter.

    As a security measure, no globals such as process or require are accessible from within the function body. Note that the function does not provide a safe sandbox and even though globals are not easily accessible, malicious code may still execute in the main process via prototype manipulation. Therefore you should only use this function to execute sanitized or safe code.

    Custom context may also be provided using the context parameter. To improve security, make sure to only pass the really necessary objects to the context. Preferably making secured copies beforehand.

    Parameters

    Returns CompiledScriptFunction

inheritedenqueueLinksByClickingElements

  • The function finds elements matching a specific CSS selector in a Puppeteer page, clicks all those elements using a mouse move and a left mouse button click and intercepts all the navigation requests that are subsequently produced by the page. The intercepted requests, including their methods, headers and payloads are then enqueued to a provided RequestQueue. This is useful to crawl JavaScript heavy pages where links are not available in href elements, but rather navigations are triggered in click handlers. If you're looking to find URLs in href attributes of the page, see enqueueLinks.

    Optionally, the function allows you to filter the target links' URLs using an array of glob or regexp patterns.

    IMPORTANT: To be able to do this, this function uses various mutations on the page, such as changing the Z-index of elements being clicked and their visibility. Therefore, it is recommended to only use this function as the last operation in the page.

    USING HEADFUL BROWSER: When using a headful browser, this function will only be able to click elements in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.

    PERFORMANCE: Clicking elements with a mouse and intercepting requests is not a low level operation that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting the scope of the clicking as much as possible by using a specific selector that targets only the elements that you assume or know will produce a navigation. You can certainly click everything by using the * selector, but be prepared to wait minutes to get results on a large and complex page.

    Example usage

    async requestHandler({ enqueueLinksByClickingElements }) {
        await enqueueLinksByClickingElements({
            selector: 'a.product-detail',
            include: [
                'https://www.example.com/handbags/**',
                'https://www.example.com/purses/**',
            ],
        });
    });

    Parameters

    Returns Promise<BatchAddRequestsResult>

    Promise that resolves to BatchAddRequestsResult object.

inheritedextendTimeout

  • extendTimeout(secs): void

  • Gives the current request secs more seconds to finish, for when how long it needs is only apparent once it is already running - a listing page that turns out to have far more to scroll through than usual, say. Prefer requestHandlerTimeoutSecs, or a per-route override via router.addHandler, whenever the time needed is known up front.

    router.addHandler('LIST', async ({ extendTimeout, page }) => {
        const pageCount = await countPages(page);
        extendTimeout(pageCount * 10);
        await scrapeAllPages(page);
    });

    Extends the request handler's own timeout and the crawler's internal one together, so the extension is not immediately undone by the latter. Calling it from a handler that has already timed out does nothing.

    Parameters

    • secs: number

    Returns void

inheritedinfiniteScroll

  • infiniteScroll(options): Promise<void>

  • Scrolls to the bottom of a page, or until it times out. Loads dynamic content when it hits the bottom of a page, and then continues scrolling.

    Parameters

    Returns Promise<void>

inheritedinjectFile

  • injectFile(filePath, options): Promise<unknown>

  • Injects a JavaScript file into current page. Unlike Puppeteer's addScriptTag function, this function works on pages with arbitrary Cross-Origin Resource Sharing (CORS) policies.

    File contents are cached for up to 10 files to limit file system access.

    Parameters

    Returns Promise<unknown>

inheritedinjectJQuery

  • injectJQuery(): Promise<unknown>

  • Injects the jQuery library into current page. jQuery is often useful for various web scraping and crawling tasks. For example, it can help extract text from HTML elements using CSS selectors.

    Beware that the injected jQuery object will be set to the window.$ variable and thus it might cause conflicts with other libraries included by the page that use the same variable name (e.g. another version of jQuery). This can affect functionality of page's scripts.

    The injected jQuery will survive page navigations and reloads.

    Example usage:

    async requestHandler({ page, injectJQuery }) {
        await injectJQuery();
        const title = await page.evaluate(() => {
            return $('head title').text();
        });
    });

    Note that injectJQuery() does not affect the Puppeteer's page.$() function in any way.

    Returns Promise<unknown>

inheritedparseWithCheerio

  • parseWithCheerio(selector, timeoutMs): Promise<CheerioAPI>

  • Returns Cheerio handle for page.content(), allowing to work with the data same way as with CheerioCrawler. When provided with the selector argument, it waits for it to be available first.

    Example usage:

    async requestHandler({ parseWithCheerio }) {
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • optionalselector: string
    • optionaltimeoutMs: number

    Returns Promise<CheerioAPI>

inheritedpushData

  • pushData(data, datasetIdentifier): Promise<void>

  • This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

    Shortcut for crawler.pushData().

    Parameters

    Returns Promise<void>

inheritedregisterDeferredCleanup

  • registerDeferredCleanup(cleanup): void

  • Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.

    The callback runs outside the request's storage transaction, so storage writes made here are applied immediately and are not rolled back when the request fails. In AdaptivePlaywrightCrawler it also runs once per request handler attempt, so a write here can land more than once for a single request. Push results from the request handler itself.

    Parameters

    • cleanup: () => Promise<unknown>

      Returns void

    inheritedremoveInterceptRequestHandler

    • removeInterceptRequestHandler(handler): Promise<void>

    • Removes request interception handler for given page.

      Parameters

      Returns Promise<void>

    inheritedsaveSnapshot

    • saveSnapshot(options): Promise<void>

    • Saves a full screenshot and HTML of the current page into a Key-Value store.

      Parameters

      Returns Promise<void>

    inheritedwaitForSelector

    • waitForSelector(selector, timeoutMs): Promise<void>

    • Wait for an element matching the selector to appear. Timeout defaults to 5s.

      Example usage:

      async requestHandler({ waitForSelector, parseWithCheerio }) {
          await waitForSelector('article h1');
          const $ = await parseWithCheerio();
          const title = $('title').text();
      });

      Parameters

      • selector: string
      • optionaltimeoutMs: number

      Returns Promise<void>

    Page Options