Changelog
All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.
3.11.5 (2024-10-04)
Note: Version bump only for package @crawlee/utils
3.11.4 (2024-09-23)
Bug Fixes
SitemapRequestList.teardown()
doesn't breakpersistState
calls (#2673) (fb2c5cd), closes /github.com/apify/crawlee/blob/f3eb99d9fa9a7aa0ec1dcb9773e666a9ac14fb76/packages/core/src/storages/sitemap_request_list.ts#L446 #2672
3.11.3 (2024-09-03)
Bug Fixes
3.11.2 (2024-08-28)
Bug Fixes
Features
3.11.1 (2024-07-24)
Bug Fixes
3.11.0 (2024-07-09)
Features
3.10.5 (2024-06-12)
Note: Version bump only for package @crawlee/utils
3.10.4 (2024-06-11)
Note: Version bump only for package @crawlee/utils
3.10.3 (2024-06-07)
Bug Fixes
3.10.2 (2024-06-03)
Bug Fixes
Features
3.10.1 (2024-05-23)
Bug Fixes
- adjust
URL_NO_COMMAS_REGEX
regexp to allow single character hostnames (#2492) (ec802e8), closes #2487
3.10.0 (2024-05-16)
Bug Fixes
- malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
- return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
- sitemap
content-type
check breaks oncontent-type
parameters (#2442) (db7d372)
Features
3.9.2 (2024-04-17)
Features
3.9.1 (2024-04-11)
Note: Version bump only for package @crawlee/utils
3.9.0 (2024-04-10)
Bug Fixes
Features
3.8.2 (2024-03-21)
Bug Fixes
3.8.1 (2024-02-22)
Note: Version bump only for package @crawlee/utils
3.8.0 (2024-02-21)
Features
- add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
- core: add
userAgent
parameter toRobotsFile.isAllowed()
+RobotsFile.from()
helper (#2338) (343c159) - Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)
3.7.3 (2024-01-30)
Bug Fixes
3.7.2 (2024-01-09)
Note: Version bump only for package @crawlee/utils
3.7.1 (2024-01-02)
Bug Fixes
3.7.0 (2023-12-21)
Bug Fixes
Features
3.6.2 (2023-11-26)
Note: Version bump only for package @crawlee/utils
3.6.1 (2023-11-15)
Note: Version bump only for package @crawlee/utils
3.6.0 (2023-11-15)
Features
3.5.8 (2023-10-17)
Bug Fixes
3.5.7 (2023-10-05)
Note: Version bump only for package @crawlee/utils
3.5.6 (2023-10-04)
Features
- add incapsula iframe selector to the blocked list (#2111) (2b17d8a), closes apify/store-website-content-crawler#154
3.5.5 (2023-10-02)
Note: Version bump only for package @crawlee/utils
3.5.4 (2023-09-11)
Note: Version bump only for package @crawlee/utils
3.5.3 (2023-08-31)
Bug Fixes
3.5.2 (2023-08-21)
Note: Version bump only for package @crawlee/utils
3.5.1 (2023-08-16)
Note: Version bump only for package @crawlee/utils
3.5.0 (2023-07-31)
Features
3.4.2 (2023-07-19)
Features
3.4.1 (2023-07-13)
Note: Version bump only for package @crawlee/utils
3.4.0 (2023-06-12)
Note: Version bump only for package @crawlee/utils
3.3.3 (2023-05-31)
Note: Version bump only for package @crawlee/utils
3.3.2 (2023-05-11)
Note: Version bump only for package @crawlee/utils
3.3.1 (2023-04-11)
Bug Fixes
- jsdom: delay closing of the window and add some polyfills (2e81618)
3.3.0 (2023-03-09)
Bug Fixes
3.2.2 (2023-02-08)
Note: Version bump only for package @crawlee/utils
3.2.1 (2023-02-07)
Note: Version bump only for package @crawlee/utils
3.2.0 (2023-02-07)
Bug Fixes
3.1.2 (2022-11-15)
Note: Version bump only for package @crawlee/utils
3.1.1 (2022-11-07)
Note: Version bump only for package @crawlee/utils
3.1.0 (2022-10-13)
Note: Version bump only for package @crawlee/utils
3.0.4 (2022-08-22)
Note: Version bump only for package @crawlee/utils