Skip to main content
Version: 3.9

Changelog

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

3.11.5 (2024-10-04)

Note: Version bump only for package @crawlee/utils

3.11.4 (2024-09-23)

Bug Fixes

3.11.3 (2024-09-03)

Bug Fixes

  • improve FACEBOOK_REGEX to match older style page URLs (#2650) (a005e69), closes #2216

3.11.2 (2024-08-28)

Bug Fixes

  • use namespace imports for cheerio to be compatible with v1 (#2641) (f48296f)

Features

3.11.1 (2024-07-24)

Bug Fixes

3.11.0 (2024-07-09)

Features

  • Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Note: Version bump only for package @crawlee/utils

3.10.4 (2024-06-11)

Note: Version bump only for package @crawlee/utils

3.10.3 (2024-06-07)

Bug Fixes

  • respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)

3.10.2 (2024-06-03)

Bug Fixes

Features

3.10.1 (2024-05-23)

Bug Fixes

  • adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487

3.10.0 (2024-05-16)

Bug Fixes

  • malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
  • return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
  • sitemap content-type check breaks on content-type parameters (#2442) (db7d372)

Features

  • implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280

3.9.2 (2024-04-17)

Features

3.9.1 (2024-04-11)

Note: Version bump only for package @crawlee/utils

3.9.0 (2024-04-10)

Bug Fixes

Features

  • expand #shadow-root elements automatically in parseWithCheerio helper (#2396) (a05b3a9)

3.8.2 (2024-03-21)

Bug Fixes

  • correctly report gzip decompression errors (#2368) (84a2f17)

3.8.1 (2024-02-22)

Note: Version bump only for package @crawlee/utils

3.8.0 (2024-02-21)

Features

  • add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
  • core: add userAgent parameter to RobotsFile.isAllowed() + RobotsFile.from() helper (#2338) (343c159)
  • Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)

3.7.3 (2024-01-30)

Bug Fixes

3.7.2 (2024-01-09)

Note: Version bump only for package @crawlee/utils

3.7.1 (2024-01-02)

Bug Fixes

  • ES2022 build compatibility and move to NodeNext for module (#2258) (7fe1e68), closes #2257

3.7.0 (2023-12-21)

Bug Fixes

  • retryOnBlocked doesn't override the blocked HTTP codes (#2243) (81672c3)

Features

3.6.2 (2023-11-26)

Note: Version bump only for package @crawlee/utils

3.6.1 (2023-11-15)

Note: Version bump only for package @crawlee/utils

3.6.0 (2023-11-15)

Features

3.5.8 (2023-10-17)

Bug Fixes

  • refactor extractUrls to split the text line by line first (#2122) (7265cd7)

3.5.7 (2023-10-05)

Note: Version bump only for package @crawlee/utils

3.5.6 (2023-10-04)

Features

3.5.5 (2023-10-02)

Note: Version bump only for package @crawlee/utils

3.5.4 (2023-09-11)

Note: Version bump only for package @crawlee/utils

3.5.3 (2023-08-31)

Bug Fixes

3.5.2 (2023-08-21)

Note: Version bump only for package @crawlee/utils

3.5.1 (2023-08-16)

Note: Version bump only for package @crawlee/utils

3.5.0 (2023-07-31)

Features

3.4.2 (2023-07-19)

Features

3.4.1 (2023-07-13)

Note: Version bump only for package @crawlee/utils

3.4.0 (2023-06-12)

Note: Version bump only for package @crawlee/utils

3.3.3 (2023-05-31)

Note: Version bump only for package @crawlee/utils

3.3.2 (2023-05-11)

Note: Version bump only for package @crawlee/utils

3.3.1 (2023-04-11)

Bug Fixes

  • jsdom: delay closing of the window and add some polyfills (2e81618)

3.3.0 (2023-03-09)

Bug Fixes

  • add proxyUrl to DownloadListOfUrlsOptions (779be1e), closes #1780

3.2.2 (2023-02-08)

Note: Version bump only for package @crawlee/utils

3.2.1 (2023-02-07)

Note: Version bump only for package @crawlee/utils

3.2.0 (2023-02-07)

Bug Fixes

  • utils: add missing dependency on ow (bf0e03c), closes #1716

3.1.2 (2022-11-15)

Note: Version bump only for package @crawlee/utils

3.1.1 (2022-11-07)

Note: Version bump only for package @crawlee/utils

3.1.0 (2022-10-13)

Note: Version bump only for package @crawlee/utils

3.0.4 (2022-08-22)

Note: Version bump only for package @crawlee/utils