BasicCrawlerOptions <Context, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>
Hierarchy
- BasicCrawlerOptions
Index
Properties
- additionalHttpErrorStatusCodes
- blockedStatusCodes
- concurrencySystem
- configuration
- contextPipelineBuilder
- errorHandler
- eventManager
- extendContext
- failedRequestHandler
- httpClient
- id
- ignoreHttpErrorStatusCodes
- initialConcurrency
- keepAlive
- logger
- maxConcurrency
- maxCrawlDepth
- maxRequestRetries
- maxRequestsPerCrawl
- maxRequestsPerMinute
- minConcurrency
- onSkippedRequest
- proxyConfiguration
- requestHandler
- requestHandlerTimeoutSecs
- requestList
- requestManager
- requestQueue
- respectRobotsTxtFile
- retryOnBlocked
- sameDomainDelaySecs
- sessionPool
- statistics
- statusMessageCallback
- statusMessageLoggingInterval
- storageBackend
- taskLoopOptions
- transactionalStorage
Properties
optionaladditionalHttpErrorStatusCodes
optionalblockedStatusCodes
HTTP status codes that indicate the session should be retired.
A 429 from a domain covered by a ThrottlingRequestManager is handled as a rate limit before this is consulted, so removing 429 here only affects domains that manager does not cover.
optionalconcurrencySystem
A pre-configured concurrency governor — the component that decides whether there is free compute for one more
task. Typically a ConcurrencySystem, though any IConcurrencySystem is accepted. All
scaling configuration (min/max/desired concurrency, scaling ratios, maxTasksPerMinute, snapshotter tuning)
lives on the instance itself.
Inject the same instance into several concurrent crawlers to cap their combined concurrency against a single budget. Each crawler still builds and drives its own AutoscaledPool; only the load/scaling accounting is shared.
Mutually exclusive with the minConcurrency/maxConcurrency/initialConcurrency/maxRequestsPerMinute
shortcuts, which configure the default system this one replaces — combining the two throws.
You own a supplied system's lifecycle: start() it before run() (which throws otherwise) and stop() it once
every crawler borrowing it has finished. The crawler does neither on your behalf.
optionalconfiguration
Custom configuration to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalcontextPipelineBuilder
Intended for BasicCrawler subclasses. Prepares a context pipeline that transforms the initial crawling context into the shape given by the Context type parameter.
The option is not required if your crawler subclass does not extend the crawling context with custom information or helpers.
Type declaration
Returns ContextPipeline<CrawlingContext<Dictionary>, Context>
optionalerrorHandler
User-provided function that allows modifying the request object before it gets retried by the crawler.
It's executed before each retry for the requests that failed less than maxRequestRetries times.
The function receives the BasicCrawlingContext as the first argument,
where the request corresponds to the request to be retried.
Second argument is the Error instance that
represents the last error thrown during processing of the request.
optionaleventManager
Custom event manager to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalextendContext
Allows the user to extend the crawling context with custom functionality (helpers, references, etc.).
extendContext runs before navigation, so the returned members are visible to the
preNavigationHooks, postNavigationHooks, and the requestHandler alike. As a consequence,
the context passed to extendContext is the pre-navigation CrawlingContext and does
not include navigation-dependent members (e.g. page, response, $, body). If you need
those, use a postNavigationHook or the requestHandler instead.
Example usage:
import { BasicCrawler } from 'crawlee';
// Create a crawler instance
const crawler = new BasicCrawler({
extendContext(context) => ({
async customHelper() {
await context.pushData({ url: context.request.url })
}
}),
async requestHandler(context) {
await context.customHelper();
},
});
Type declaration
Parameters
context: CrawlingContext<Dictionary>
Returns Awaitable<ContextExtension>
optionalfailedRequestHandler
A function to handle requests that failed more than maxRequestRetries times.
The function receives the BasicCrawlingContext as the first argument,
where the request corresponds to the failed request.
Second argument is the Error instance that
represents the last error thrown during processing of the request.
optionalhttpClient
HTTP client implementation for the sendRequest context helper and for plain HTTP crawling.
Defaults to ImpitHttpClient when @crawlee/impit-client is installed, otherwise FetchHttpClient.
optionalid
A unique identifier for the crawler instance. This ID is used to isolate the state returned by
crawler.useState() from other crawler instances.
When multiple crawler instances use useState() without an explicit id, they will share the same
state object for backward compatibility. A warning will be logged in this case.
To ensure each crawler has its own isolated state that also persists across script restarts (e.g., during Apify migrations), provide a stable, unique ID for each crawler instance.
optionalignoreHttpErrorStatusCodes
An array of HTTP response Status Codes to be excluded from error consideration. By default, status codes >= 500 trigger errors.
optionalinitialConcurrency
Sets the concurrency (parallelism) the crawl starts with, before any scaling happens. Shortcut for the
desiredConcurrency option of the crawler's default
ConcurrencySystem. Defaults to minConcurrency.
optionalkeepAlive
Allows to keep the crawler alive even if the RequestQueue gets empty.
By default, the crawler.run() will resolve once the queue is empty. With keepAlive: true it will keep running,
waiting for more requests to come. Use crawler.stop() to exit the crawler gracefully, or crawler.teardown() to stop it immediately.
optionallogger
Custom logger to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalmaxConcurrency
Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the
maxConcurrency option of the crawler's default
ConcurrencySystem.
optionalmaxCrawlDepth
Maximum depth of the crawl. If not set, the crawl will continue until all requests are processed.
Setting this to 0 will only process the initial requests, skipping all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests.
Passing 1 will process the initial requests and all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests in the handler for initial requests.
optionalmaxRequestRetries
Specifies the maximum number of retries allowed for a request if its processing fails.
This includes retries due to navigation errors, session/proxy errors, or errors thrown from user-supplied
functions (requestHandler, preNavigationHooks, postNavigationHooks).
optionalmaxRequestsPerCrawl
Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers.
NOTE: In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
optionalmaxRequestsPerMinute
The maximum number of requests per minute the crawler should run.
By default, this is set to Infinity, but we can pass any positive, non-zero integer.
Shortcut for the maxTasksPerMinute option of the
crawler's default ConcurrencySystem.
optionalminConcurrency
Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the
minConcurrency option of the crawler's default
ConcurrencySystem.
WARNING: If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically.
optionalonSkippedRequest
When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped
- based on robots.txt file,
- because they don't match enqueueLinks filters,
- because they are redirected to a URL that doesn't match the enqueueLinks strategy,
- or because the
maxRequestsPerCrawllimit has been reached
optionalproxyConfiguration
If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration.
optionalrequestHandler
User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
The function receives the BasicCrawlingContext as an argument,
where the request represents the URL to crawl.
The function must return a promise, which is then awaited by the crawler.
If the function throws an exception, the crawler will try to re-crawl the
request later, up to the maxRequestRetries times.
If all the retries fail, the crawler calls the function
provided to the failedRequestHandler parameter.
To make this work, we should always
let our function throw exceptions rather than catch them.
The exceptions are logged to the request using the
Request.pushErrorMessage() function.
optionalrequestHandlerTimeoutSecs
Timeout in which the function passed as requestHandler needs to finish, in seconds.
optionalrequestList
Static list of URLs to be processed.
Deprecated - Use the requestManager option instead. To combine a read-only loader (such as a RequestList)
with a writable queue, build a tandem with requestList.toTandem(requestQueue)
and pass the result as requestManager. When both requestList and requestQueue are provided, they are
combined into a tandem automatically.
optionalrequestManager
Manager of requests that should be processed by the crawler. Mutually exclusive with the deprecated
requestQueue and requestList options.
If not provided, the crawler will open the default RequestQueue when it is first needed.
optionalrequestQueue
Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
Deprecated - Use the requestManager option instead. A RequestQueue is itself a request manager, so you can
pass it directly as requestManager.
optionalrespectRobotsTxtFile
If set to true, the crawler will automatically try to fetch the robots.txt file for each domain,
and skip those that are not allowed. This also prevents disallowed URLs to be added via enqueueLinks.
If an object is provided, it may contain a userAgent property to specify which user-agent
should be used when checking the robots.txt file. If not provided, the default user-agent * will be used.
optionalretryOnBlocked
If set to true, the crawler will automatically try to bypass any detected bot protection.
Currently supports:
optionalsameDomainDelaySecs
Indicates how much time (in seconds) to wait before crawling another same domain request. Subdomains are paced together with the site they belong to.
Offered to the crawler's request manager as a minIntervalEverywhere PacingSignal; a manager that
already paces every domain it dispatches to takes it, so no domain ends up with two clocks. Otherwise the
crawler wraps its request manager in a ThrottlingRequestManager of its own.
optionalsessionPool
An existing session pool instance to use. When provided, the crawler will use this pool directly instead of creating a new one, enabling session sharing across multiple crawlers. The crawler will not tear down a shared pool — the caller is responsible for its lifecycle.
Accepts the built-in SessionPool or any object implementing the ISessionPool interface, so custom session-management strategies can be plugged in.
optionalstatistics
A preconfigured statistics instance. When provided, the crawler records into it instead of building its own and
will not reset() it between run() calls. Accepts the built-in Statistics or any object
implementing IStatistics.
Custom fields declared via stateExtension are carried over to
crawler.statistics.state:
const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });
const crawler = new BasicCrawler({
statistics,
requestHandler: async () => {
statistics.state.productsFound++;
},
});
await crawler.run();
// the custom fields are typed on `crawler.statistics` too
console.log(crawler.statistics.state.productsFound);
optionalstatusMessageCallback
Allows overriding the default status message. The callback needs to call crawler.setStatusMessage() explicitly.
The default status message is provided in the parameters.
const crawler = new CheerioCrawler({
statusMessageCallback: async (ctx) => {
return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG'
},
statusMessageLoggingInterval: 1, // defaults to 10s
async requestHandler({ $, enqueueLinks, request, log }) {
// ...
},
});
optionalstatusMessageLoggingInterval
Defines the length of the interval for calling the setStatusMessage in seconds.
optionalstorageBackend
Custom storage backend to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionaltaskLoopOptions
Lets you override the predicates that steer the crawler's task loop: isTaskReadyFunction (may another request
start?) and isFinishedFunction (is the crawl over?). The task itself — fetching a request and running it
through the pipeline — is owned by the crawler and cannot be overridden.
Concurrency is configured elsewhere — through the minConcurrency/maxConcurrency/maxRequestsPerMinute
shortcuts, or a concurrencySystem for finer control.
optionaltransactionalStorage
Makes the storage writes performed while handling a request atomic with respect to the request succeeding: they are recorded in a StorageTransaction spanning the whole request lifecycle and only applied when the request handler succeeds, so a thrown handler leaves no partial writes behind and a retry does not double-write. Reads within the handler see its own writes.
false disables the mechanism entirely; an object overrides the per-storage-type
StorageWritePolicy (e.g. { requestQueue: 'deferred' } for all-or-nothing enqueues).
withDirectStorageAccess is the per-call-site escape hatch; useState() is deliberately
not transactional.
An array of additional HTTP response Status Codes to be treated as errors. By default, status codes >= 500 trigger errors.