ConcurrencySystem
Implements
Index
Constructors
constructor
Parameters
options: ConcurrencySystemOptions = {}
Returns ConcurrencySystem
Accessors
currentConcurrency
The number of parallel tasks currently booked against this governor, regardless of which pool booked them.
Returns number
desiredConcurrency
Gets the desired concurrency for the system, which is an estimated number of parallel tasks that the system can currently support.
Returns number
Sets the desired concurrency for the system, i.e. the number of tasks that should be running in parallel if there's large enough supply of tasks.
Parameters
value: number
Returns void
isRunning
Whether the system is currently monitoring load and autoscaling the budget.
Returns boolean
maxConcurrency
Gets the maximum number of tasks running in parallel.
Returns number
Sets the maximum number of tasks running in parallel. Lowering it below the current
desiredConcurrencypulls that down to the new ceiling too, so the change takes effect immediately (in-flight tasks are never cancelled — the budget simply drains to the new limit as they settle).Parameters
value: number
Returns void
minConcurrency
Gets the minimum number of tasks running in parallel.
Returns number
Sets the minimum number of tasks running in parallel.
WARNING: If you set this value too high with respect to the available system memory and CPU, your code might run extremely slow or crash. If you're not sure, just keep the default value and the concurrency will scale up automatically.
Parameters
value: number
Returns void
Methods
[asyncDispose]
Returns Promise<void>
getCurrentStatus
What the system currently makes of the machine: the per-signal overload verdicts, evaluated over the task-gating window, exactly as
hasCapacityForTask()sees them. The one public window into load monitoring — useful for answering why a crawl is not scaling up.Returns SystemInfo
hasCapacityForTask
May one more task start right now? Returns
falsewhen the shared budget is spent (desired concurrency reached) or when the machine is overloaded pastminConcurrency.One budget for the whole machine, so the asking consumer is ignored — and therefore optional here, unlike in the interface, letting the answer be queried directly.
Parameters
optional_consumer: ConcurrencyConsumer
Returns boolean
registerTaskEnd
Returns a slot to the shared budget, whoever booked it.
Parameters
optional_consumer: ConcurrencyConsumer
Returns void
start
Boots the underlying snapshotter and the autoscaling interval. Idempotent, so a shared system isn't restarted when handed to another consumer; concurrent callers await one startup. Rejects, leaving nothing running, if a signal fails to start.
Returns Promise<void>
stop
Stops the snapshotter and intervals. Idempotent and safe to call even if the system was never started.
Returns Promise<void>
tryRegisterTaskStart
Atomically books a task against the shared budget: re-checks
hasCapacityForTask()plus the per-minute task cap and increments the current concurrency in one synchronous step, returningfalse(without booking) when there is no room. Call right before the task actually runs.The cap is enforced here rather than in the pre-check so that an empty queue never blocks the pool for a whole extra minute.
Parameters
optionalconsumer: ConcurrencyConsumer
Returns boolean
The shareable "governor" behind an AutoscaledPool: it decides whether there is free compute for one more task by combining live system load (via an internal Snapshotter) with a concurrency budget it autoscales over time.
Sharing one instance between several pools (and therefore several crawlers) caps their combined compute, instead of letting each scale independently and oversubscribe the machine.
Whoever builds the instance owns its lifecycle: call
start()before any borrowing pool runs andstop()once they are all done (crawlers do this for the default system they build, never for an injected one). Both calls are idempotent, and the firststop()tears the system down for every borrower.