JSDOMCrawler guide
JSDOMCrawler
is very useful for scraping with the Window API.
How the crawler works
JSDOMCrawler
crawls by making plain HTTP requests to the provided URLs using the specialized got-scraping HTTP client. The URLs are fed to the crawler using RequestQueue
. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the additionalMimeTypes
option.
Modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards, and together they create the final page. To crawl those, see PuppeteerCrawler
and PlaywrightCrawler
.
Once the page's HTML is retrieved, the crawler will pass it to JSDOM for parsing. The result is a window
property, which should be familiar to frontend developers. You can use the Window API to do all sorts of lookups and manipulation of the page's HTML, but in scraping, you will mostly use it to find specific HTML elements and extract their data.
Example use of browser JavaScript:
// Return the page title
document.title; // browsers
window.document.title; // JSDOM
When to use JSDOMCrawler
JSDOMCrawler
really shines when CheerioCrawler
is just not enough. There is an entire set of APIs available!
Advantages:
- Easy to set up
- Familiar for frontend developers
- Content can be manipulated
- Automatically avoids some anti-scraping bans
Disadvantages:
- Slower than
CheerioCrawler
- Does not work for websites that require JavaScript rendering
- May easily overload the target website with requests
Example use of Element API
Find all links on a page
This snippet finds all <a>
elements which have the href
attribute and extracts the hrefs into an array.
Array.from(document.querySelectorAll('a[href]')).map((a) => a.href);
Other examples
Visit the Examples section to browse examples of JSDOMCrawler
usage. Almost all examples show JSDOMCrawler
code in their code tabs.