Skip to main content
Version: 3.10

CheerioCrawlingContext <UserData, JSONData>


  • InternalHttpCrawlingContext<UserData, JSONData, CheerioCrawler>
    • CheerioCrawlingContext




$: CheerioAPI

The Cheerio object with parsed HTML. Cheerio is available only for HTML and XML content types.


addRequests: (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>) => Promise<void>

Add requests directly to the request queue.

Type declaration

    • (requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[], options?: ReadonlyObjectDeep<RequestQueueOperationOptions>): Promise<void>
    • Parameters

      • requestsLike: readonly (string | ReadonlyObjectDeep<Partial<RequestOptions<Dictionary>> & { regex?: RegExp; requestsFromUrl?: string }> | ReadonlyObjectDeep<Request<Dictionary>>)[]
      • optionaloptions: ReadonlyObjectDeep<RequestQueueOperationOptions>

        Options for the request queue

      Returns Promise<void>


body: string | Buffer

The request body of the web page. The type depends on the Content-Type header of the web page:

  • String for text/html, application/xhtml+xml, application/xml MIME content types
  • Buffer for others MIME content types


contentType: { encoding: BufferEncoding; type: string }

Parsed Content-Type header: { type, encoding }.

Type declaration

  • encoding: BufferEncoding
  • type: string



getKeyValueStore: (idOrName?: string) => Promise<KeyValueStore>

Get a key-value store with given name or id, or the default one for the crawler.

Type declaration


id: string


json: JSONData

The parsed object from JSON string if the response contains the content type application/json.


log: Log

A preconfigured logger for the request handler.


proxyInfo?: ProxyInfo

An object with information about currently used proxy by the crawler and configured by the ProxyConfiguration class.


request: Request<UserData>

The original Request object.


response: PlainResponse


session?: Session


useState: <State>(defaultValue?: State) => Promise<State>

Returns the state - a piece of mutable persistent data shared across all the request handler runs.

Type declaration

    • <State>(defaultValue?: State): Promise<State>
    • Type parameters

      • State: Dictionary = Dictionary


      • optionaldefaultValue: State

      Returns Promise<State>



  • This function automatically finds and enqueues links from the current page, adding them to the RequestQueue currently used by the crawler.

    Optionally, the function allows you to filter the target links' URLs using an array of globs or regular expressions and override settings of the enqueued Request objects.

    Check out the Crawl a website with relative links example for more details regarding its usage.

    Example usage

    async requestHandler({ enqueueLinks }) {
    await enqueueLinks({
    globs: [


    Returns Promise<BatchAddRequestsResult>

    Promise that resolves to BatchAddRequestsResult object.


  • parseWithCheerio(): Promise<CheerioAPI>
  • Returns Cheerio handle, this is here to unify the crawler API, so they all have this handy method. It has the same return type as the $ context property, use it only if you are abstracting your workflow to support different context types in one handler.

    Example usage:

    async requestHandler({ parseWithCheerio }) {
    const $ = await parseWithCheerio();
    const title = $('title').text();

    Returns Promise<CheerioAPI>


  • pushData(data?: ReadonlyDeep<Dictionary | Dictionary[]>, datasetIdOrName?: string): Promise<void>
  • This function allows you to push data to a Dataset specified by name, or the one currently used by the crawler.

    Shortcut for crawler.pushData().


    • optionaldata: ReadonlyDeep<Dictionary | Dictionary[]>

      Data to be pushed to the default dataset.

    • optionaldatasetIdOrName: string

    Returns Promise<void>


  • sendRequest<Response>(overrideOptions?: Partial<OptionsInit>): Promise<Response<Response>>
  • Fires HTTP request via got-scraping, allowing to override the request options on the fly.

    This is handy when you work with a browser crawler but want to execute some requests outside it (e.g. API requests). Check the Skipping navigations for certain requests example for more detailed explanation of how to do that.

    async requestHandler({ sendRequest }) {
    const { body } = await sendRequest({
    // override headers only
    headers: { ... },

    Type parameters

    • Response = string


    • optionaloverrideOptions: Partial<OptionsInit>

    Returns Promise<Response<Response>>