DOMParser <Parsed>

Turns a response body into a DOM representation and knows how to query it. Passing one to DOMCrawler is what makes the crawler jsdom-based, linkedom-based, or based on a DOM implementation of your own.

Example usage:

import { DOMCrawler } from 'crawlee';
import type { DOMParser } from 'crawlee';

const myParser: DOMParser<{ body: string }> = {
    // ...
};

const crawler = new DOMCrawler({
    parser: myParser,
    async requestHandler({ body }) {
        // ...
    },
});

Index

Properties

• mutable
• placeholderMembers

Methods

• cleanup
• extractLinks
• parse
• select
• toCheerio

Properties

optionalreadonlymutable

mutable?: boolean

Whether the parse result can change after parse returned - the case when the DOM implementation runs the page scripts. If it can, waitForSelector polls until the timeout elapses; otherwise it fails as soon as the selector does not match.

readonlyplaceholderMembers

placeholderMembers: Record<keyof Parsed & string, true>

The context members parse contributes, mapped to true. Used to build the placeholders that report a helpful error when the members are accessed after skipNavigation - the Record type forces every key of Parsed to be listed, so the compiler catches an omission that would otherwise yield undefined (rather than throwing) after skipNavigation.

Methods

optionalcleanup

• cleanup(parsed): Awaitable<void>

• Releases whatever parse allocated. Called after the request handler finishes or fails, and skipped entirely when navigation was skipped.

  ---

  Parameters

  • parsed: Parsed

  Returns Awaitable<void>

extractLinks

• extractLinks(parsed, selector, baseUrl): Awaitable<string[]>

• Returns the URLs the selector matches, resolved against baseUrl.

  ---

  Parameters

  • parsed: Parsed

  • selector: string

  • baseUrl: string

  Returns Awaitable<string[]>

parse

• parse(context): Awaitable<Parsed>

• Parameters

  • context: InternalHttpCrawlingContext<any, any>

  Returns Awaitable<Parsed>

select

• select(parsed, selector): Awaitable<ArrayLike<unknown>>

• Returns the current matches of selector. Only the count is used, by waitForSelector.

  ---

  Parameters

  • parsed: Parsed

  • selector: string

  Returns Awaitable<ArrayLike<unknown>>

optionaltoCheerio

• toCheerio(parsed): Awaitable<CheerioAPI>

• Returns a Cheerio handle over the parse result, for parsers that are backed by Cheerio anyway. Without it, parseWithCheerio parses body again.

  ---

  Parameters

  • parsed: Parsed

  Returns Awaitable<CheerioAPI>