BasicCrawler <Context>
Hierarchy
- BasicCrawler
- BrowserCrawler
- HttpCrawler
Index
Constructors
Properties
Methods
Constructors
constructor
All
BasicCrawler
parameters are passed via an options object.Type parameters
- Context: CrawlingContext<Dictionary<any>, Context> = BasicCrawlingContext<Dictionary<any>>
Parameters
options: BasicCrawlerOptions<Context> = {}
config: Configuration = ...
Returns BasicCrawler<Context>
Properties
optionalautoscaledPool
A reference to the underlying AutoscaledPool class that manages the concurrency of the crawler.
NOTE: This property is only initialized after calling the
crawler.run()
function. We can use it to change the concurrency settings on the fly, to pause the crawler by callingautoscaledPool.pause()
or to abort it by callingautoscaledPool.abort()
.
readonlyconfig
optionalrequestList
A reference to the underlying RequestList class that manages the crawler's requests. Only available if used by the crawler.
optionalrequestQueue
Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites. A reference to the underlying RequestQueue class that manages the crawler's requests. Only available if used by the crawler.
readonlyrouter
Default Router instance that will be used if we don't specify any requestHandler
.
See router.addHandler()
and router.addDefaultHandler()
.
optionalsessionPool
A reference to the underlying SessionPool class that manages the crawler's sessions. Only available if used by the crawler.
readonlystats
A reference to the underlying Statistics class that collects and logs run statistics for requests.
Methods
addRequests
Adds requests to be processed by the crawler
Parameters
requests: (string | RequestOptions<Dictionary<any>> | Request<Dictionary<any>>)[]
The requests to add
options: CrawlerAddRequestsOptions = {}
Options for the request queue
Returns Promise<CrawlerAddRequestsResult>
getRequestQueue
Returns Promise<RequestQueue>
run
Runs the crawler. Returns a promise that gets resolved once all the requests are processed. We can use the
requests
parameter to enqueue the initial requests - it is a shortcut for runningcrawler.addRequests()
before thecrawler.run()
.Parameters
optionalrequests: (string | RequestOptions<Dictionary<any>> | Request<Dictionary<any>>)[]
The requests to add
optionaloptions: CrawlerAddRequestsOptions
Options for the request queue
Returns Promise<FinalStatistics>
useState
Type parameters
- State: Dictionary<any> = Dictionary<any>
Parameters
defaultValue: State = ...
Returns Promise<State>
Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
BasicCrawler
is a low-level tool that requires the user to implement the page download and data extraction functionality themselves. If we want a crawler that already facilitates this functionality, we should consider using CheerioCrawler, PuppeteerCrawler or PlaywrightCrawler.BasicCrawler
invokes the user-providedrequestHandler
for each Request object, which represents a single URL to crawl. The Request objects are fed from the RequestList or RequestQueue instances provided by therequestList
orrequestQueue
constructor options, respectively. If neitherrequestList
norrequestQueue
options are provided, the crawler will open the default request queue either when thecrawler.addRequests()
function is called, or ifrequests
parameter (representing the initial requests) of thecrawler.run()
function is provided.If both
requestList
andrequestQueue
options are used, the instance first processes URLs from the RequestList and automatically enqueues all of them to the RequestQueue before it starts their processing. This ensures that a single URL is not crawled multiple times.The crawler finishes if there are no more Request objects to crawl.
New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the AutoscaledPool class. All AutoscaledPool configuration options can be passed to the
autoscaledPoolOptions
parameter of theBasicCrawler
constructor. For user convenience, theminConcurrency
andmaxConcurrency
options of the underlying AutoscaledPool constructor are available directly in theBasicCrawler
constructor.Example usage: