@crawlee/basic
Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.
BasicCrawler
is a low-level tool that requires the user to implement the page download and data extraction functionality themselves.
If we want a crawler that already facilitates this functionality, we should consider using CheerioCrawler, PuppeteerCrawler or PlaywrightCrawler.
BasicCrawler
invokes the user-provided requestHandler
for each Request object, which represents a single URL to crawl. The Request objects are fed from the RequestList or RequestQueue instances provided by the requestList
or requestQueue
constructor options, respectively. If neither requestList
nor requestQueue
options are provided, the crawler will open the default request queue either when the crawler.addRequests()
function is called, or if requests
parameter (representing the initial requests) of the crawler.run()
function is provided.
If both requestList
and requestQueue
options are used, the instance first processes URLs from the RequestList and automatically enqueues all of them to the RequestQueue before it starts their processing. This ensures that a single URL is not crawled multiple times.
The crawler finishes if there are no more Request objects to crawl.
New requests are only dispatched when there is enough free CPU and memory available, using the functionality provided by the AutoscaledPool class. All AutoscaledPool configuration options can be passed to the autoscaledPoolOptions
parameter of the BasicCrawler
constructor. For user convenience, the minConcurrency
and maxConcurrency
options of the underlying AutoscaledPool constructor are available directly in the BasicCrawler
constructor.
Example usage
import { BasicCrawler, Dataset } from 'crawlee';
// Create a crawler instance
const crawler = new BasicCrawler({
async requestHandler({ request, sendRequest }) {
// 'request' contains an instance of the Request class
// Here we simply fetch the HTML of the page and store it to a dataset
const { body } = await sendRequest({
url: request.url,
method: request.method,
body: request.payload,
headers: request.headers,
});
await Dataset.pushData({
url: request.url,
html: body,
})
},
});
// Enqueue the initial requests and run the crawler
await crawler.run([
'http://www.example.com/page-1',
'http://www.example.com/page-2',
]);
Index
Crawlers
Other
- AddRequestsBatchedOptions
- AddRequestsBatchedResult
- AutoscaledPool
- AutoscaledPoolOptions
- BLOCKED_STATUS_CODES
- BaseHttpClient
- BaseHttpResponseData
- ClientInfo
- Configuration
- ConfigurationOptions
- Cookie
- CrawlingContext
- CreateSession
- CriticalError
- Dataset
- DatasetConsumer
- DatasetContent
- DatasetDataOptions
- DatasetExportOptions
- DatasetExportToOptions
- DatasetIteratorOptions
- DatasetMapper
- DatasetOptions
- DatasetReducer
- EnqueueLinksOptions
- EnqueueStrategy
- ErrnoException
- ErrorSnapshotter
- ErrorTracker
- ErrorTrackerOptions
- EventManager
- EventType
- EventTypeName
- FinalStatistics
- GetUserDataFromRequest
- GlobInput
- GlobObject
- GotScrapingHttpClient
- HttpRequest
- HttpRequestOptions
- HttpResponse
- IRequestList
- IStorage
- KeyConsumer
- KeyValueStore
- KeyValueStoreIteratorOptions
- KeyValueStoreOptions
- LoadedRequest
- LocalEventManager
- Log
- LogLevel
- Logger
- LoggerJson
- LoggerOptions
- LoggerText
- MAX_POOL_SIZE
- NonRetryableError
- PERSIST_STATE_KEY
- PersistenceOptions
- ProxyConfiguration
- ProxyConfigurationFunction
- ProxyConfigurationOptions
- ProxyInfo
- PseudoUrl
- PseudoUrlInput
- PseudoUrlObject
- PushErrorMessageOptions
- QueueOperationInfo
- RecordOptions
- RedirectHandler
- RegExpInput
- RegExpObject
- Request
- RequestHandlerResult
- RequestList
- RequestListOptions
- RequestListSourcesFunction
- RequestListState
- RequestOptions
- RequestProvider
- RequestProviderOptions
- RequestQueue
- RequestQueueOperationOptions
- RequestQueueOptions
- RequestQueueV1
- RequestQueueV2
- RequestState
- RequestTransform
- ResponseLike
- ResponseTypes
- RestrictedCrawlingContext
- RetryRequestError
- Router
- RouterHandler
- RouterRoutes
- Session
- SessionError
- SessionOptions
- SessionPool
- SessionPoolOptions
- SessionState
- SitemapRequestList
- SitemapRequestListOptions
- SnapshotResult
- Snapshotter
- SnapshotterOptions
- Source
- StatisticPersistedState
- StatisticState
- Statistics
- StatisticsOptions
- StorageClient
- StorageManagerOptions
- StreamingHttpResponse
- SystemInfo
- SystemStatus
- SystemStatusOptions
- TieredProxy
- UrlPatternObject
- UseStateOptions
- checkStorageAccess
- enqueueLinks
- filterRequestsByPatterns
- log
- processHttpRequestOptions
- purgeDefaultStorages
- tryAbsoluteURL
- useState
- withCheckedStorageAccess
- BasicCrawlerOptions
- BasicCrawlingContext
- CrawlerAddRequestsOptions
- CrawlerAddRequestsResult
- CrawlerExperiments
- CrawlerRunOptions
- CreateContextOptions
- StatusMessageCallbackParams
- ErrorHandler
- RequestHandler
- StatusMessageCallback
- BASIC_CRAWLER_TIMEOUT_BUFFER_SECS
- createBasicRouter
Other
AddRequestsBatchedOptions
AddRequestsBatchedResult
AutoscaledPool
AutoscaledPoolOptions
BLOCKED_STATUS_CODES
BaseHttpClient
BaseHttpResponseData
ClientInfo
Configuration
ConfigurationOptions
Cookie
CrawlingContext
CreateSession
CriticalError
Dataset
DatasetConsumer
DatasetContent
DatasetDataOptions
DatasetExportOptions
DatasetExportToOptions
DatasetIteratorOptions
DatasetMapper
DatasetOptions
DatasetReducer
EnqueueLinksOptions
EnqueueStrategy
ErrnoException
ErrorSnapshotter
ErrorTracker
ErrorTrackerOptions
EventManager
EventType
EventTypeName
FinalStatistics
GetUserDataFromRequest
GlobInput
GlobObject
GotScrapingHttpClient
HttpRequest
HttpRequestOptions
HttpResponse
IRequestList
IStorage
KeyConsumer
KeyValueStore
KeyValueStoreIteratorOptions
KeyValueStoreOptions
LoadedRequest
LocalEventManager
Log
LogLevel
Logger
LoggerJson
LoggerOptions
LoggerText
MAX_POOL_SIZE
NonRetryableError
PERSIST_STATE_KEY
PersistenceOptions
ProxyConfiguration
ProxyConfigurationFunction
ProxyConfigurationOptions
ProxyInfo
PseudoUrl
PseudoUrlInput
PseudoUrlObject
PushErrorMessageOptions
QueueOperationInfo
RecordOptions
RedirectHandler
RegExpInput
RegExpObject
Request
RequestHandlerResult
RequestList
RequestListOptions
RequestListSourcesFunction
RequestListState
RequestOptions
RequestProvider
RequestProviderOptions
RequestQueue
RequestQueueOperationOptions
RequestQueueOptions
RequestQueueV1
RequestQueueV2
RequestState
RequestTransform
ResponseLike
ResponseTypes
RestrictedCrawlingContext
RetryRequestError
Router
RouterHandler
RouterRoutes
Session
SessionError
SessionOptions
SessionPool
SessionPoolOptions
SessionState
SitemapRequestList
SitemapRequestListOptions
SnapshotResult
Snapshotter
SnapshotterOptions
Source
StatisticPersistedState
StatisticState
Statistics
StatisticsOptions
StorageClient
StorageManagerOptions
StreamingHttpResponse
SystemInfo
SystemStatus
SystemStatusOptions
TieredProxy
UrlPatternObject
UseStateOptions
checkStorageAccess
enqueueLinks
filterRequestsByPatterns
log
processHttpRequestOptions
purgeDefaultStorages
tryAbsoluteURL
useState
withCheckedStorageAccess
ErrorHandler
Type parameters
- Context: CrawlingContext = LoadedContext<BasicCrawlingContext & RestrictedCrawlingContext>
Type declaration
Parameters
inputs: LoadedContext<Context>
error: Error
Returns Awaitable<void>
RequestHandler
Type parameters
- Context: CrawlingContext = LoadedContext<BasicCrawlingContext & RestrictedCrawlingContext>
Type declaration
Parameters
inputs: LoadedContext<Context>
Returns Awaitable<void>
StatusMessageCallback
Type parameters
- Context: CrawlingContext = BasicCrawlingContext
- Crawler: BasicCrawler<any> = BasicCrawler<Context>
Type declaration
Parameters
params: StatusMessageCallbackParams<Context, Crawler>
Returns Awaitable<void>
Additional number of seconds used in CheerioCrawler and BrowserCrawler to set a reasonable
requestHandlerTimeoutSecs
for BasicCrawler that would not impare functionality (not timeout before crawlers).