@crawlee/puppeteer
Provides a simple framework for parallel crawling of web pages using headless Chrome with Puppeteer. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
Since PuppeteerCrawler uses headless Chrome to download web pages and extract data, it is useful for crawling of websites that require to execute JavaScript. If the target website doesn't need JavaScript, consider using CheerioCrawler, which downloads the pages using raw HTTP requests and is about 10x faster.
The source URLs are represented using Request objects that are fed from RequestList or RequestQueue instances provided by the PuppeteerCrawlerOptions.requestList or PuppeteerCrawlerOptions.requestQueue constructor options, respectively.
If both PuppeteerCrawlerOptions.requestList and PuppeteerCrawlerOptions.requestQueue are used, the instance first processes URLs from the RequestList and automatically enqueues all of them to RequestQueue before it starts their processing. This ensures that a single URL is not crawled multiple times.
The crawler finishes when there are no more Request objects to crawl.
PuppeteerCrawler opens a new Chrome page (i.e. tab) for each Request object to crawl
and then calls the function provided by user as the PuppeteerCrawlerOptions.requestHandler option.
New pages are only opened when there is enough free CPU and memory available,
using the functionality provided by the AutoscaledPool class.
All AutoscaledPool configuration options can be passed to the PuppeteerCrawlerOptions.autoscaledPoolOptions
parameter of the PuppeteerCrawler constructor. For user convenience, the minConcurrency and maxConcurrency
AutoscaledPoolOptions are available directly in the PuppeteerCrawler constructor.
Note that the pool of Puppeteer instances is internally managed by the BrowserPool class.
Example usage
const crawler = new PuppeteerCrawler({
async requestHandler({ page, request }) {
// This function is called to extract data from a single web page
// 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called
// 'request' is an instance of Request class with information about the page to load
await Dataset.pushData({
title: await page.title(),
url: request.url,
succeeded: true,
})
},
async failedRequestHandler({ request }) {
// This function is called when the crawling of a request failed too many times
await Dataset.pushData({
url: request.url,
succeeded: false,
errors: request.errorMessages,
})
},
});
await crawler.run([
'http://www.example.com/page-1',
'http://www.example.com/page-2',
]);
Index
References
- AutoscaledPool
- AutoscaledPoolOptions
- BASIC_CRAWLER_TIMEOUT_BUFFER_SECS
- BasicCrawler
- BasicCrawlerOptions
- BasicCrawlingContext
- BlockRequestsOptions
- BrowserCrawler
- BrowserCrawlerOptions
- BrowserCrawlingContext
- BrowserErrorHandler
- BrowserHook
- BrowserLaunchContext
- BrowserRequestHandler
- ClientInfo
- CompiledScriptFunction
- CompiledScriptParams
- Configuration
- ConfigurationOptions
- Cookie
- CrawlerAddRequestsOptions
- CrawlerAddRequestsResult
- CrawlingContext
- CreateContextOptions
- CreateSession
- CriticalError
- Dataset
- DatasetConsumer
- DatasetContent
- DatasetDataOptions
- DatasetIteratorOptions
- DatasetMapper
- DatasetOptions
- DatasetReducer
- EnqueueLinksByClickingElementsOptions
- EnqueueLinksOptions
- EnqueueStrategy
- ErrorHandler
- EventManager
- EventType
- EventTypeName
- ExportOptions
- FinalStatistics
- GlobInput
- GlobObject
- IStorage
- InfiniteScrollOptions
- InjectFileOptions
- InterceptHandler
- KeyConsumer
- KeyValueStore
- KeyValueStoreIteratorOptions
- KeyValueStoreOptions
- LocalEventManager
- Log
- LogLevel
- Logger
- LoggerJson
- LoggerOptions
- LoggerText
- NonRetryableError
- ProxyConfiguration
- ProxyConfigurationFunction
- ProxyConfigurationOptions
- ProxyInfo
- PseudoUrl
- PseudoUrlInput
- PseudoUrlObject
- PuppeteerDirectNavigationOptions
- PushErrorMessageOptions
- QueueOperationInfo
- QueueOperationInfoOptions
- RecordOptions
- RegExpInput
- RegExpObject
- Request
- RequestHandler
- RequestList
- RequestListOptions
- RequestListSourcesFunction
- RequestListState
- RequestOptions
- RequestQueue
- RequestQueueOperationOptions
- RequestQueueOptions
- RequestTransform
- RetryRequestError
- Router
- RouterHandler
- SaveSnapshotOptions
- Session
- SessionOptions
- SessionPool
- SessionPoolOptions
- SessionState
- Snapshotter
- SnapshotterOptions
- Source
- StatisticPersistedState
- StatisticState
- Statistics
- StorageClient
- StorageManagerOptions
- SystemInfo
- SystemStatus
- SystemStatusOptions
- UrlPatternObject
- UseStateOptions
- createBasicRouter
- enqueueLinks
- filterRequestsByPatterns
- log
- purgeDefaultStorages
- useState
Namespaces
Crawlers
Interfaces
Type Aliases
Functions
References
AutoscaledPool
Re-exports AutoscaledPool
AutoscaledPoolOptions
Re-exports AutoscaledPoolOptions
BASIC_CRAWLER_TIMEOUT_BUFFER_SECS
Re-exports BASIC_CRAWLER_TIMEOUT_BUFFER_SECS
BasicCrawler
Re-exports BasicCrawler
BasicCrawlerOptions
Re-exports BasicCrawlerOptions
BasicCrawlingContext
Re-exports BasicCrawlingContext
BlockRequestsOptions
Re-exports BlockRequestsOptions
BrowserCrawler
Re-exports BrowserCrawler
BrowserCrawlerOptions
Re-exports BrowserCrawlerOptions
BrowserCrawlingContext
Re-exports BrowserCrawlingContext
BrowserErrorHandler
Re-exports BrowserErrorHandler
BrowserHook
Re-exports BrowserHook
BrowserLaunchContext
Re-exports BrowserLaunchContext
BrowserRequestHandler
Re-exports BrowserRequestHandler
ClientInfo
Re-exports ClientInfo
CompiledScriptFunction
Re-exports CompiledScriptFunction
CompiledScriptParams
Re-exports CompiledScriptParams
Configuration
Re-exports Configuration
ConfigurationOptions
Re-exports ConfigurationOptions
Cookie
Re-exports Cookie
CrawlerAddRequestsOptions
Re-exports CrawlerAddRequestsOptions
CrawlerAddRequestsResult
Re-exports CrawlerAddRequestsResult
CrawlingContext
Re-exports CrawlingContext
CreateContextOptions
Re-exports CreateContextOptions
CreateSession
Re-exports CreateSession
CriticalError
Re-exports CriticalError
Dataset
Re-exports Dataset
DatasetConsumer
Re-exports DatasetConsumer
DatasetContent
Re-exports DatasetContent
DatasetDataOptions
Re-exports DatasetDataOptions
DatasetIteratorOptions
Re-exports DatasetIteratorOptions
DatasetMapper
Re-exports DatasetMapper
DatasetOptions
Re-exports DatasetOptions
DatasetReducer
Re-exports DatasetReducer
EnqueueLinksByClickingElementsOptions
Re-exports EnqueueLinksByClickingElementsOptions
EnqueueLinksOptions
Re-exports EnqueueLinksOptions
EnqueueStrategy
Re-exports EnqueueStrategy
ErrorHandler
Re-exports ErrorHandler
EventManager
Re-exports EventManager
EventType
Re-exports EventType
EventTypeName
Re-exports EventTypeName
ExportOptions
Re-exports ExportOptions
FinalStatistics
Re-exports FinalStatistics
GlobInput
Re-exports GlobInput
GlobObject
Re-exports GlobObject
IStorage
Re-exports IStorage
InfiniteScrollOptions
Re-exports InfiniteScrollOptions
InjectFileOptions
Re-exports InjectFileOptions
InterceptHandler
Re-exports InterceptHandler
KeyConsumer
Re-exports KeyConsumer
KeyValueStore
Re-exports KeyValueStore
KeyValueStoreIteratorOptions
Re-exports KeyValueStoreIteratorOptions
KeyValueStoreOptions
Re-exports KeyValueStoreOptions
LocalEventManager
Re-exports LocalEventManager
Log
Re-exports Log
LogLevel
Re-exports LogLevel
Logger
Re-exports Logger
LoggerJson
Re-exports LoggerJson
LoggerOptions
Re-exports LoggerOptions
LoggerText
Re-exports LoggerText
NonRetryableError
Re-exports NonRetryableError
ProxyConfiguration
Re-exports ProxyConfiguration
ProxyConfigurationFunction
Re-exports ProxyConfigurationFunction
ProxyConfigurationOptions
Re-exports ProxyConfigurationOptions
ProxyInfo
Re-exports ProxyInfo
PseudoUrl
Re-exports PseudoUrl
PseudoUrlInput
Re-exports PseudoUrlInput
PseudoUrlObject
Re-exports PseudoUrlObject
PuppeteerDirectNavigationOptions
Renames and re-exports DirectNavigationOptions
PushErrorMessageOptions
Re-exports PushErrorMessageOptions
QueueOperationInfo
Re-exports QueueOperationInfo
QueueOperationInfoOptions
Re-exports QueueOperationInfoOptions
RecordOptions
Re-exports RecordOptions
RegExpInput
Re-exports RegExpInput
RegExpObject
Re-exports RegExpObject
Request
Re-exports Request
RequestHandler
Re-exports RequestHandler
RequestList
Re-exports RequestList
RequestListOptions
Re-exports RequestListOptions
RequestListSourcesFunction
Re-exports RequestListSourcesFunction
RequestListState
Re-exports RequestListState
RequestOptions
Re-exports RequestOptions
RequestQueue
Re-exports RequestQueue
RequestQueueOperationOptions
Re-exports RequestQueueOperationOptions
RequestQueueOptions
Re-exports RequestQueueOptions
RequestTransform
Re-exports RequestTransform
RetryRequestError
Re-exports RetryRequestError
Router
Re-exports Router
RouterHandler
Re-exports RouterHandler
SaveSnapshotOptions
Re-exports SaveSnapshotOptions
Session
Re-exports Session
SessionOptions
Re-exports SessionOptions
SessionPool
Re-exports SessionPool
SessionPoolOptions
Re-exports SessionPoolOptions
SessionState
Re-exports SessionState
Snapshotter
Re-exports Snapshotter
SnapshotterOptions
Re-exports SnapshotterOptions
Source
Re-exports Source
StatisticPersistedState
Re-exports StatisticPersistedState
StatisticState
Re-exports StatisticState
Statistics
Re-exports Statistics
StorageClient
Re-exports StorageClient
StorageManagerOptions
Re-exports StorageManagerOptions
SystemInfo
Re-exports SystemInfo
SystemStatus
Re-exports SystemStatus
SystemStatusOptions
Re-exports SystemStatusOptions
UrlPatternObject
Re-exports UrlPatternObject
UseStateOptions
Re-exports UseStateOptions
createBasicRouter
Re-exports createBasicRouter
enqueueLinks
Re-exports enqueueLinks
filterRequestsByPatterns
Re-exports filterRequestsByPatterns
log
Re-exports log
purgeDefaultStorages
Re-exports purgeDefaultStorages
useState
Re-exports useState