CrawlerAddRequestsOptions

Hierarchy

• AddRequestsBatchedOptions
  • CrawlerAddRequestsOptions
    • CrawlerRunOptions

Index

Properties

• batchSize
• forefront
• waitBetweenBatchesMillis
• waitForAllRequestsToBeAdded

Properties

optionalinheritedbatchSize

batchSize?: number = 1000

Inherited from AddRequestsBatchedOptions.batchSize

optionalinheritedforefront

forefront?: boolean = false

Inherited from AddRequestsBatchedOptions.forefront

If set to true:

• while adding the request to the queue: the request will be added to the foremost position in the queue.
• while reclaiming the request: the request will be placed to the beginning of the queue, so that it's returned in the next call to RequestQueue.fetchNextRequest. By default, it's put to the end of the queue.

In case the request is already present in the queue, this option has no effect.

If more requests are added with this option at once, their order in the following fetchNextRequest call is arbitrary.

optionalinheritedwaitBetweenBatchesMillis

waitBetweenBatchesMillis?: number = 1000

Inherited from AddRequestsBatchedOptions.waitBetweenBatchesMillis

optionalinheritedwaitForAllRequestsToBeAdded

waitForAllRequestsToBeAdded?: boolean = false

Inherited from AddRequestsBatchedOptions.waitForAllRequestsToBeAdded

Whether to wait for all the provided requests to be added, instead of waiting just for the initial batch of up to batchSize.