Changelog
All notable changes to this project will be documented in this file. See Conventional Commits for commit guidelines.
3.11.1 (2024-07-24)
Note: Version bump only for package @crawlee/core
3.11.0 (2024-07-09)
Features
3.10.5 (2024-06-12)
Bug Fixes
3.10.4 (2024-06-11)
Bug Fixes
- add
waitForAllRequestsToBeAdded
option toenqueueLinks
helper (925546b), closes #2318 - respect
crawler.log
when creating child logger forStatistics
(0a0d75d), closes #2412
3.10.3 (2024-06-07)
Bug Fixes
- respect implicit router when no
requestHandler
is provided inAdaptiveCrawler
(#2518) (31083aa) - revert the scaling steps back to 5% (5bf32f8)
Features
3.10.2 (2024-06-03)
Note: Version bump only for package @crawlee/core
3.10.1 (2024-05-23)
Bug Fixes
- investigate and temp fix for possible 0-concurrency bug in RQv2 (#2494) (4ebe820)
- provide URLs to the error snapshot (#2482) (7f64145), closes /github.com/apify/apify-sdk-js/blob/master/packages/apify/src/key_value_store.ts#L25
3.10.0 (2024-05-16)
Bug Fixes
EnqueueStrategy.All
erroring with links using unsupported protocols (#2389) (8db3908)- core: conversion between tough cookies and browser pool cookies (#2443) (74f73ab)
- core: fire local
SystemInfo
events every second (#2454) (1fa9a66) - core: use createSessionFunction when loading Session from persisted state (#2444) (3c56b4c)
- double tier decrement in tiered proxy (#2468) (3a8204b)
Features
- implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280
- make
RequestQueue
v2 the default queue, see more on Apify blog (#2390) (41ae8ab), closes #2388
Performance Improvements
- improve scaling based on memory (#2459) (2d5d443)
- optimize
RequestList
memory footprint (#2466) (12210bd) - optimize adding large amount of requests via
crawler.addRequests()
(#2456) (6da86a8)
3.9.2 (2024-04-17)
Bug Fixes
3.9.1 (2024-04-11)
Note: Version bump only for package @crawlee/core
3.9.0 (2024-04-10)
Bug Fixes
- include actual key in error message of KVS'
setValue
(#2411) (9089bf1) - notify autoscaled pool about newly added requests (#2400) (a90177d)
Features
createAdaptivePlaywrightRouter
utility (#2415) (cee4778), closes #2407tieredProxyUrls
for ProxyConfiguration (#2348) (5408c7f)- better
newUrlFunction
for ProxyConfiguration (#2392) (330598b), closes #2348 #2065
3.8.2 (2024-03-21)
Bug Fixes
- core: solve possible dead locks in
RequestQueueV2
(#2376) (ffba095) - use 0 (number) instead of false as default for sessionRotationCount (#2372) (667a3e7)
Features
- implement global storage access checking and use it to prevent unwanted side effects in adaptive crawler (#2371) (fb3b7da), closes #2364
3.8.1 (2024-02-22)
Bug Fixes
3.8.0 (2024-02-21)
Bug Fixes
Features
KeyValueStore.recordExists()
(#2339) (8507a65)- accessing crawler state, key-value store and named datasets via crawling context (#2283) (58dd5fc)
- adaptive playwright crawler (#2316) (8e4218a)
3.7.3 (2024-01-30)
Bug Fixes
3.7.2 (2024-01-09)
Bug Fixes
3.7.1 (2024-01-02)
Note: Version bump only for package @crawlee/core
3.7.0 (2023-12-21)
Bug Fixes
retryOnBlocked
doesn't override the blocked HTTP codes (#2243) (81672c3)- filter out empty globs (#2205) (41322ab), closes #2200
- make SessionPool queue up getSession calls to prevent overruns (#2239) (0f5665c), closes #1667
Features
- allow configuring crawler statistics (#2213) (9fd60e4), closes #1789
- check enqueue link strategy post redirect (#2238) (3c5f9d6), closes #2173