config: Configuration = ...
Gets the the number of parallel tasks currently running in the pool.
Gets the desired concurrency for the pool, which is an estimated number of parallel tasks that the system can currently support.
Sets the desired concurrency for the pool, i.e. the number of tasks that should be running in parallel if there’s large enough supply of tasks.
Gets the maximum number of tasks running in parallel.
Sets the maximum number of tasks running in parallel.
Gets the minimum number of tasks running in parallel.
Sets the minimum number of tasks running in parallel.
WARNING: If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash. If you’re not sure, just keep the default value and the concurrency will scale up automatically.
Aborts the run of the auto-scaled pool and destroys it. The promise returned from the AutoscaledPool.run function will immediately resolve, no more new tasks will be spawned and all running tasks will be left in their current state.
Due to the nature of the tasks, auto-scaled pool cannot reliably guarantee abortion of all the running tasks, therefore, no abortion is attempted and some of the tasks may finish, while others may not. Essentially, auto-scaled pool doesn’t care about their state after the invocation of
.abort(), but that does not mean that some parts of their asynchronous chains of commands will not execute.
Prevents the auto-scaled pool from starting new tasks, but allows the running ones to finish (unlike abort, which terminates them). Used together with AutoscaledPool.resume
The function’s promise will resolve once all running tasks have completed and the pool is effectively idle. If the
timeoutSecsargument is provided, the promise will reject with a timeout error after the
The promise returned from the AutoscaledPool.run function will not resolve when
.pause()is invoked (unlike abort, which resolves it).
Resumes the operation of the autoscaled-pool by allowing more tasks to be run. Used together with AutoscaledPool.pause
Tasks will automatically start running again in
Runs the auto-scaled pool. Returns a promise that gets resolved or rejected once all the tasks are finished or one of them fails.
The information about the CPU and memory usage is obtained by the Snapshotter class, which makes regular snapshots of system resources that may be either local or from the Apify cloud infrastructure in case the process is running on the Apify platform. Meaningful data gathered from these snapshots is provided to
AutoscaledPoolby the SystemStatus class.
Before running the pool, you need to implement the following three functions: AutoscaledPoolOptions.runTaskFunction, AutoscaledPoolOptions.isTaskReadyFunction and AutoscaledPoolOptions.isFinishedFunction.
The auto-scaled pool is started by calling the AutoscaledPool.run function. The pool periodically queries the AutoscaledPoolOptions.isTaskReadyFunction function for more tasks, managing optimal concurrency, until the function resolves to
false. The pool then queries the AutoscaledPoolOptions.isFinishedFunction. If it resolves to
true, the run finishes after all running tasks complete. If it resolves to
false, it assumes there will be more tasks available later and keeps periodically querying for tasks. If any of the tasks throws then the AutoscaledPool.run function rejects the promise with an error.
The pool evaluates whether it should start a new task every time one of the tasks finishes and also in the interval set by the