Skip to main content
Version: Next

InternalHttpCrawlingContext <UserData, JSONData>

Hierarchy

Index

Properties

Methods

Properties

inheritedaddRequests

addRequests: (requestsLike, options) => Promise<AddRequestsBatchedResult>

Add requests directly to the request queue currently used by the crawler.

Optionally, the function allows you to filter the target URLs using an array of glob or regexp patterns, the same way enqueueLinks does for extracted links.

Type declaration

body

body: string | Buffer<ArrayBufferLike>

The request body of the web page. The type depends on the Content-Type header of the web page:

  • String for text/html, application/xhtml+xml, application/xml MIME content types
  • Buffer for others MIME content types

contentType

contentType: { encoding: BufferEncoding; type: string }

Parsed Content-Type header: { type, encoding }.

Type declaration

  • encoding: BufferEncoding
  • type: string

inheritedgetKeyValueStore

getKeyValueStore: (identifier) => Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

Get a key-value store with given name or id, or the default one for the crawler.

Type declaration

    • (identifier): Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

    • Parameters

      Returns Promise<Pick<KeyValueStore, id | name | getValue | getAutoSavedValue | setValue | getPublicUrl>>

inheritedid

id: string

json

json: JSONData

The parsed object from JSON string if the response contains the content type application/json.

inheritedlog

log: CrawleeLogger

A preconfigured logger for the request handler.

optionalinheritedproxyInfo

proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.

inheritedrequest

request: LoadedRequest<CrawleeRequest<UserData>>

The request object that was successfully loaded and navigated to, including the loadedUrl property.

inheritedresponse

response: Response

The HTTP response object containing status code, headers, and other response metadata.

inheritedsendRequest

sendRequest: (requestOverrides, optionsOverrides) => Promise<Response>

Fires HTTP request via the internal HTTP client, allowing to override the request options on the fly.

This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.

async requestHandler({ sendRequest }) {
    const { body } = await sendRequest({
        // override headers only
        headers: { ... },
    });
},

Type declaration

    • (requestOverrides, optionsOverrides): Promise<Response>

inheritedsession

session: ISession

inheriteduseState

useState: <State>(defaultValue) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.

Type declaration

    • <State>(defaultValue): Promise<State>

    • Parameters

      • optionaldefaultValue: State

      Returns Promise<State>

Methods

inheritedextendTimeout

  • extendTimeout(secs): void

  • Gives the current request secs more seconds to finish, for when how long it needs is only apparent once it is already running - a listing page that turns out to have far more to scroll through than usual, say. Prefer requestHandlerTimeoutSecs, or a per-route override via router.addHandler, whenever the time needed is known up front.

    router.addHandler('LIST', async ({ extendTimeout, page }) => {
        const pageCount = await countPages(page);
        extendTimeout(pageCount * 10);
        await scrapeAllPages(page);
    });

    Extends the request handler's own timeout and the crawler's internal one together, so the extension is not immediately undone by the latter. Calling it from a handler that has already timed out does nothing.

    Parameters

    • secs: number

    Returns void

parseWithCheerio

  • parseWithCheerio(selector, timeoutMs): Promise<CheerioAPI>

  • Returns Cheerio handle for page.content(), allowing to work with the data same way as with CheerioCrawler. When provided with the selector argument, it will throw if it's not available.

    Example usage:

    async requestHandler({ parseWithCheerio }) {
        const $ = await parseWithCheerio();
        const title = $('title').text();
    });

    Parameters

    • optionalselector: string
    • optionaltimeoutMs: number

    Returns Promise<CheerioAPI>

inheritedpushData

  • pushData(data, datasetIdentifier): Promise<void>

  • This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

    Shortcut for crawler.pushData().

    Parameters

    Returns Promise<void>

inheritedregisterDeferredCleanup

  • registerDeferredCleanup(cleanup): void

  • Register a function to be called at the very end of the request handling process. This is useful for resources that should be accessible to error handlers, for instance.

    The callback runs outside the request's storage transaction, so storage writes made here are applied immediately and are not rolled back when the request fails. In AdaptivePlaywrightCrawler it also runs once per request handler attempt, so a write here can land more than once for a single request. Push results from the request handler itself.

    Parameters

    • cleanup: () => Promise<unknown>

      Returns void

    waitForSelector

    • waitForSelector(selector, timeoutMs): Promise<void>

    • Wait for an element matching the selector to appear. Timeout is ignored.

      Example usage:

      async requestHandler({ waitForSelector, parseWithCheerio }) {
          await waitForSelector('article h1');
          const $ = await parseWithCheerio();
          const title = $('title').text();
      });

      Parameters

      • selector: string
      • optionaltimeoutMs: number

      Returns Promise<void>

    Page Options