Skip to main content
Version: Next

AdaptivePlaywrightCrawlerContext

Hierarchy

Index

Properties

Methods

Properties

inheritedaddRequests

addRequests: (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>) => Promise<void>

Add requests directly to the request queue.

Type declaration

    • (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>): Promise<void>

    • Parameters

      • requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[]
      • optionaloptions: ReadonlyObjectDeep<RequestQueueOperationOptions>

        Options for the request queue

      Returns Promise<void>

inheritedenqueueLinks

enqueueLinks: (options?: ReadonlyObjectDeep<Omit<EnqueueLinksOptions, requestQueue>>) => Promise<unknown>

This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.

Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.

Check out the Crawl a website with relative links example for more details regarding its usage.

Example usage

async requestHandler({ enqueueLinks }) {
    await enqueueLinks({
      globs: [
          'https://www.example.com/handbags/*',
      ],
    });
},

Type declaration

    • Parameters

      • optionaloptions: ReadonlyObjectDeep<Omit<EnqueueLinksOptions, requestQueue>>

        All enqueueLinks() parameters are passed via an options object.

      Returns Promise<unknown>

inheritedgetKeyValueStore

getKeyValueStore: (idOrName?: string) => Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue>>

Get a key-value store with given name or id, or the default one for the crawler.

Type declaration

    • (idOrName?: string): Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue>>

    • Parameters

      • optionalidOrName: string

      Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue>>

inheritedid

id: string

inheritedlog

log: Log

A preconfigured logger for the request handler.

optionalinheritedproxyInfo

proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.

inheritedrequest

request: Request<Dictionary>

The original Request object.

optionalinheritedsession

session?: Session

inheriteduseState

useState: (defaultValue?: State) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.

Type declaration

    • (defaultValue?: State): Promise<State>

    • Parameters

      • optionaldefaultValue: State

      Returns Promise<State>

Methods

parseWithCheerio

  • parseWithCheerio(selector?: string, timeoutMs?: number): Promise<CheerioAPI>

  • Returns Cheerio handle for page.content(), allowing to work with the data same way as with CheerioCrawler. When provided with the selector argument, it will first look for the selector with a 5s timeout.

    Example usage:

    async requestHandler({ parseWithCheerio }) {
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • optionalselector: string
    • optionaltimeoutMs: number

    Returns Promise<CheerioAPI>

inheritedpushData

  • pushData(data?: ReadonlyDeep<Dictionary | Dictionary[]>, datasetIdOrName?: string): Promise<void>

  • This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

    Shortcut for crawler.pushData().

    Parameters

    • optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>

      Data to be pushed to the default dataset.

    • optionaldatasetIdOrName: string

    Returns Promise<void>

querySelector

  • querySelector(selector: string, timeoutMs?: number): Promise<Cheerio<Element>>

  • Wait for an element matching the selector to appear and return a Cheerio object of matched elements. Timeout defaults to 5s.

    Parameters

    • selector: string
    • optionaltimeoutMs: number

    Returns Promise<Cheerio<Element>>

waitForSelector

  • waitForSelector(selector: string, timeoutMs?: number): Promise<void>

  • Wait for an element matching the selector to appear. Timeout defaults to 5s.

    Example usage:

    async requestHandler({ waitForSelector, parseWithCheerio }) {
        await waitForSelector('article h1');
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • selector: string
    • optionaltimeoutMs: number

    Returns Promise<void>

Page Options