DOMCrawlingHelpers

Index

Methods

• enqueueLinks
• extractLinks
• parseWithCheerio
• waitForSelector

Methods

enqueueLinks

• enqueueLinks(options): Promise<AddRequestsBatchedResult>

• Helper function for extracting URLs from the parsed DOM and adding them to the request queue.

  ---

  Parameters

  • optionaloptions: EnqueueLinksOptions

  Returns Promise<AddRequestsBatchedResult>

extractLinks

• extractLinks(options): Promise<string[]>

• Extracts URLs from the parsed DOM, without adding them to the request queue.

  ---

  Parameters

  • optionaloptions: ExtractLinksOptions

  Returns Promise<string[]>

parseWithCheerio

• parseWithCheerio(selector, timeoutMs): Promise<CheerioAPI>

• Returns Cheerio handle, allowing to work with the data same way as with CheerioCrawler. When provided with the selector argument, it will throw if it's not available.

  Example usage:

  async requestHandler({ parseWithCheerio }) {
      const $ = await parseWithCheerio();
      const title = $('title').text();
  });

  ---

  Parameters

  • optionalselector: string

  • optionaltimeoutMs: number

  Returns Promise<CheerioAPI>

waitForSelector

• waitForSelector(selector, timeoutMs): Promise<void>

• Wait for an element matching the selector to appear. The timeoutMs only has an effect when the parser is mutable (e.g. JSDOMCrawler with runScripts: true); otherwise the selector is checked once and the call resolves or throws immediately. Timeout defaults to 5s.

  Example usage:

  async requestHandler({ waitForSelector, parseWithCheerio }) {
      await waitForSelector('article h1');
      const $ = await parseWithCheerio();
      const title = $('title').text();
  });

  ---

  Parameters

  • selector: string

  • optionaltimeoutMs: number

  Returns Promise<void>