Changelog

All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.

3.12.0 (2024-11-04)

Bug Fixes

• .trim() urls from pretty-printed sitemap.xml files (#2709) (802a6fe), closes #2698

Features

• allow using other HTTP clients (#2661) (568c655), closes #2659

3.11.5 (2024-10-04)

Note: Version bump only for package @crawlee/utils

3.11.4 (2024-09-23)

Bug Fixes

• SitemapRequestList.teardown() doesn't break persistState calls (#2673) (fb2c5cd), closes /github.com/apify/crawlee/blob/f3eb99d9fa9a7aa0ec1dcb9773e666a9ac14fb76/packages/core/src/storages/sitemap_request_list.ts#L446 #2672

3.11.3 (2024-09-03)

Bug Fixes

• improve FACEBOOK_REGEX to match older style page URLs (#2650) (a005e69), closes #2216

3.11.2 (2024-08-28)

Bug Fixes

• use namespace imports for cheerio to be compatible with v1 (#2641) (f48296f)

Features

• resilient sitemap loading (#2619) (1dd7660)

3.11.1 (2024-07-24)

Bug Fixes

• use getHTML in the shadow root expansion (#2587) (a244d62), closes #2583

3.11.0 (2024-07-09)

Features

• Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Note: Version bump only for package @crawlee/utils

3.10.4 (2024-06-11)

Note: Version bump only for package @crawlee/utils

3.10.3 (2024-06-07)

Bug Fixes

• respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)

3.10.2 (2024-06-03)

Bug Fixes

• Autodetect sitemap filetype from content (#2497) (62a9f40), closes #2461

Features

• Loading sitemaps from string (#2496) (38ed0d6), closes #2460

3.10.1 (2024-05-23)

Bug Fixes

• adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487

3.10.0 (2024-05-16)

Bug Fixes

• malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
• return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
• sitemap content-type check breaks on content-type parameters (#2442) (db7d372)

Features

• implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280

3.9.2 (2024-04-17)

Features

• sitemap: Support CDATA in sitemaps (#2424) (635f046)

3.9.1 (2024-04-11)

Note: Version bump only for package @crawlee/utils

3.9.0 (2024-04-10)

Bug Fixes

• sitemaps support application/xml (#2408) (cbcf47a)

Features

• expand #shadow-root elements automatically in parseWithCheerio helper (#2396) (a05b3a9)

3.8.2 (2024-03-21)

Bug Fixes

• correctly report gzip decompression errors (#2368) (84a2f17)

3.8.1 (2024-02-22)

Note: Version bump only for package @crawlee/utils

3.8.0 (2024-02-21)

Features

• add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
• core: add userAgent parameter to RobotsFile.isAllowed() + RobotsFile.from() helper (#2338) (343c159)
• Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)

3.7.3 (2024-01-30)

Bug Fixes

• pass on an invisible CF turnstile (#2277) (d8734e7), closes #2256

3.7.2 (2024-01-09)

Note: Version bump only for package @crawlee/utils

3.7.1 (2024-01-02)

Bug Fixes

• ES2022 build compatibility and move to NodeNext for module (#2258) (7fe1e68), closes #2257

3.7.0 (2023-12-21)

Bug Fixes

• retryOnBlocked doesn't override the blocked HTTP codes (#2243) (81672c3)

Features

• robots.txt and sitemap.xml utils (#2214) (fdfec4f), closes #2187

3.6.2 (2023-11-26)

Note: Version bump only for package @crawlee/utils

3.6.1 (2023-11-15)

Note: Version bump only for package @crawlee/utils

3.6.0 (2023-11-15)

Features

• got-scraping v4 (#2110) (2f05ed2)

3.5.8 (2023-10-17)

Bug Fixes

• refactor extractUrls to split the text line by line first (#2122) (7265cd7)

3.5.7 (2023-10-05)

Note: Version bump only for package @crawlee/utils

3.5.6 (2023-10-04)

Features

• add incapsula iframe selector to the blocked list (#2111) (2b17d8a), closes apify/store-website-content-crawler#154

3.5.5 (2023-10-02)

Note: Version bump only for package @crawlee/utils

3.5.4 (2023-09-11)

Note: Version bump only for package @crawlee/utils

3.5.3 (2023-08-31)

Bug Fixes

• pin all internal dependencies (#2041) (d6f2b17), closes #2040

3.5.2 (2023-08-21)

Note: Version bump only for package @crawlee/utils

3.5.1 (2023-08-16)

Note: Version bump only for package @crawlee/utils

3.5.0 (2023-07-31)

Features

• retire session on proxy error (#2002) (8c0928b), closes #1912

3.4.2 (2023-07-19)

Features

• retryOnBlocked detects blocked webpage (#1956) (766fa9b)

3.4.1 (2023-07-13)

Note: Version bump only for package @crawlee/utils

3.4.0 (2023-06-12)

Note: Version bump only for package @crawlee/utils

3.3.3 (2023-05-31)

Note: Version bump only for package @crawlee/utils

3.3.2 (2023-05-11)

Note: Version bump only for package @crawlee/utils

3.3.1 (2023-04-11)

Bug Fixes

• jsdom: delay closing of the window and add some polyfills (2e81618)

3.3.0 (2023-03-09)

Bug Fixes

• add proxyUrl to DownloadListOfUrlsOptions (779be1e), closes #1780

3.2.2 (2023-02-08)

Note: Version bump only for package @crawlee/utils

3.2.1 (2023-02-07)

Note: Version bump only for package @crawlee/utils

3.2.0 (2023-02-07)

Bug Fixes

• utils: add missing dependency on ow (bf0e03c), closes #1716

3.1.2 (2022-11-15)

Note: Version bump only for package @crawlee/utils

3.1.1 (2022-11-07)

Note: Version bump only for package @crawlee/utils

3.1.0 (2022-10-13)

Note: Version bump only for package @crawlee/utils

3.0.4 (2022-08-22)

Note: Version bump only for package @crawlee/utils