LinkeDOMCrawlerOptions <ContextExtension, ExtendedContext, UserData, JSONData, Routes, StatisticStateExtension>
Hierarchy
- HttpCrawlerOptions<LinkeDOMCrawlingContext<UserData, JSONData>, ContextExtension, ExtendedContext, Routes, StatisticStateExtension>
- LinkeDOMCrawlerOptions
Index
Properties
- additionalHttpErrorStatusCodes
- additionalMimeTypes
- blockedStatusCodes
- concurrencySystem
- configuration
- contextPipelineBuilder
- errorHandler
- eventManager
- extendContext
- failedRequestHandler
- forceResponseEncoding
- httpClient
- id
- ignoreHttpErrorStatusCodes
- ignoreTlsErrors
- keepAlive
- logger
- maxConcurrency
- maxCrawlDepth
- maxRequestRetries
- maxRequestsPerCrawl
- maxRequestsPerMinute
- minConcurrency
- navigationTimeoutSecs
- onSkippedRequest
- postNavigationHooks
- preNavigationHooks
- proxyConfiguration
- requestHandler
- requestHandlerTimeoutSecs
- requestList
- requestManager
- requestQueue
- respectRobotsTxtFile
- retryOnBlocked
- sameDomainDelaySecs
- saveResponseCookies
- sessionPool
- statistics
- statusMessageCallback
- statusMessageLoggingInterval
- storageBackend
- suggestResponseEncoding
- taskLoopOptions
- transactionalStorage
Properties
optionalinheritedadditionalHttpErrorStatusCodes
optionalinheritedadditionalMimeTypes
An array of MIME types
you want the crawler to load and process. By default, only text/html, application/xhtml+xml, text/xml, application/xml,
and application/json MIME types are supported.
optionalinheritedblockedStatusCodes
HTTP status codes that indicate the session should be retired.
A 429 from a domain covered by a ThrottlingRequestManager is handled as a rate limit before this is consulted, so removing 429 here only affects domains that manager does not cover.
optionalinheritedconcurrencySystem
A pre-configured concurrency governor — the component that decides whether there is free compute for one more
task. Typically a ConcurrencySystem, though any IConcurrencySystem is accepted. All
scaling configuration (min/max/desired concurrency, scaling ratios, maxTasksPerMinute, snapshotter tuning)
lives on the instance itself.
Inject the same instance into several concurrent crawlers to cap their combined concurrency against a single budget. Each crawler still builds and drives its own AutoscaledPool; only the load/scaling accounting is shared.
Mutually exclusive with the minConcurrency/maxConcurrency/maxRequestsPerMinute shortcuts, which configure
the default system this one replaces — combining the two throws.
You own a supplied system's lifecycle: start() it before run() (which throws otherwise) and stop() it once
every crawler borrowing it has finished. The crawler does neither on your behalf.
optionalinheritedconfiguration
Custom configuration to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalinheritedcontextPipelineBuilder
Intended for BasicCrawler subclasses. Prepares a context pipeline that transforms the initial crawling context into the shape given by the Context type parameter.
The option is not required if your crawler subclass does not extend the crawling context with custom information or helpers.
Type declaration
Returns ContextPipeline<CrawlingContext<Dictionary>, LinkeDOMCrawlingContext<UserData, JSONData>>
optionalinheritederrorHandler
User-provided function that allows modifying the request object before it gets retried by the crawler.
It's executed before each retry for the requests that failed less than maxRequestRetries times.
The function receives the BasicCrawlingContext as the first argument,
where the request corresponds to the request to be retried.
Second argument is the Error instance that
represents the last error thrown during processing of the request.
optionalinheritedeventManager
Custom event manager to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalinheritedextendContext
Allows the user to extend the crawling context with custom functionality (helpers, references, etc.).
extendContext runs before navigation, so the returned members are visible to the
preNavigationHooks, postNavigationHooks, and the requestHandler alike. As a consequence,
the context passed to extendContext is the pre-navigation CrawlingContext and does
not include navigation-dependent members (e.g. page, response, $, body). If you need
those, use a postNavigationHook or the requestHandler instead.
Example usage:
import { BasicCrawler } from 'crawlee';
// Create a crawler instance
const crawler = new BasicCrawler({
extendContext(context) => ({
async customHelper() {
await context.pushData({ url: context.request.url })
}
}),
async requestHandler(context) {
await context.customHelper();
},
});
Type declaration
Parameters
context: CrawlingContext<Dictionary>
Returns Awaitable<ContextExtension>
optionalinheritedfailedRequestHandler
A function to handle requests that failed more than maxRequestRetries times.
The function receives the BasicCrawlingContext as the first argument,
where the request corresponds to the failed request.
Second argument is the Error instance that
represents the last error thrown during processing of the request.
optionalinheritedforceResponseEncoding
By default this crawler will extract correct encoding from the HTTP response headers. Use forceResponseEncoding
to force a certain encoding, disregarding the response headers.
To only provide a default for missing encodings, use HttpCrawlerOptions.suggestResponseEncoding
// Will force windows-1250 encoding even if headers say otherwise
forceResponseEncoding: 'windows-1250'
optionalinheritedhttpClient
HTTP client implementation for the sendRequest context helper and for plain HTTP crawling.
Defaults to ImpitHttpClient when @crawlee/impit-client is installed, otherwise FetchHttpClient.
optionalinheritedid
A unique identifier for the crawler instance. This ID is used to isolate the state returned by
crawler.useState() from other crawler instances.
When multiple crawler instances use useState() without an explicit id, they will share the same
state object for backward compatibility. A warning will be logged in this case.
To ensure each crawler has its own isolated state that also persists across script restarts (e.g., during Apify migrations), provide a stable, unique ID for each crawler instance.
optionalinheritedignoreHttpErrorStatusCodes
An array of HTTP response Status Codes to be excluded from error consideration. By default, status codes >= 500 trigger errors.
optionalinheritedignoreTlsErrors
If set to true, TLS/SSL certificate errors are ignored. Forwarded to the HTTP client as
ignoreTlsErrors on every navigation request, so custom
BaseHttpClient implementations should honor that flag (the built-in impit and got-scraping
clients do; the native fetch fallback cannot disable TLS verification and warns instead).
optionalinheritedkeepAlive
Allows to keep the crawler alive even if the RequestQueue gets empty.
By default, the crawler.run() will resolve once the queue is empty. With keepAlive: true it will keep running,
waiting for more requests to come. Use crawler.stop() to exit the crawler gracefully, or crawler.teardown() to stop it immediately.
optionalinheritedlogger
Custom logger to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalinheritedmaxConcurrency
Sets the maximum concurrency (parallelism) for the crawl. Shortcut for the
maxConcurrency option of the crawler's default
ConcurrencySystem.
optionalinheritedmaxCrawlDepth
Maximum depth of the crawl. If not set, the crawl will continue until all requests are processed.
Setting this to 0 will only process the initial requests, skipping all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests.
Passing 1 will process the initial requests and all links enqueued by crawlingContext.enqueueLinks and crawlingContext.addRequests in the handler for initial requests.
optionalinheritedmaxRequestRetries
Specifies the maximum number of retries allowed for a request if its processing fails.
This includes retries due to navigation errors, session/proxy errors, or errors thrown from user-supplied
functions (requestHandler, preNavigationHooks, postNavigationHooks).
optionalinheritedmaxRequestsPerCrawl
Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. This value should always be set in order to prevent infinite loops in misconfigured crawlers.
NOTE: In cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
optionalinheritedmaxRequestsPerMinute
The maximum number of requests per minute the crawler should run.
By default, this is set to Infinity, but we can pass any positive, non-zero integer.
Shortcut for the maxTasksPerMinute option of the
crawler's default ConcurrencySystem.
optionalinheritedminConcurrency
Sets the minimum concurrency (parallelism) for the crawl. Shortcut for the
minConcurrency option of the crawler's default
ConcurrencySystem.
WARNING: If we set this value too high with respect to the available system memory and CPU, our crawler will run extremely slow or crash. If not sure, it's better to keep the default value and the concurrency will scale up automatically.
optionalinheritednavigationTimeoutSecs
Timeout for the whole navigation phase, given in seconds. A single window shared by the
preNavigationHooks, the navigation (the HTTP request to the resource), and the postNavigationHooks -
so a slow hook eats into the same budget the navigation uses. Separate from the
requestHandlerTimeoutSecs, which times only the
request handler.
optionalinheritedonSkippedRequest
When a request is skipped for some reason, you can use this callback to act on it. This is currently fired for requests skipped
- based on robots.txt file,
- because they don't match enqueueLinks filters,
- because they are redirected to a URL that doesn't match the enqueueLinks strategy,
- or because the
maxRequestsPerCrawllimit has been reached
optionalinheritedpostNavigationHooks
Async functions that are sequentially evaluated after the navigation. Good for checking if the navigation was successful.
The function accepts crawlingContext as the only parameter.
A hook may optionally return a partial object whose properties are merged into the crawling context,
which is useful for overriding the response after solving a challenge or re-fetching the resource.
Example:
postNavigationHooks: [
async (crawlingContext) => {
if (await needsRevalidation(crawlingContext)) {
return { response: await refetch(crawlingContext.request) };
}
},
]
optionalinheritedpreNavigationHooks
Async functions that are sequentially evaluated before the navigation. Good for setting additional cookies
or browser properties before navigation. The function accepts one parameter crawlingContext,
which is passed to the requestAsBrowser() function the crawler calls to navigate.
A hook may optionally return a partial object whose properties are merged into the crawling context, allowing the hook to override context members for subsequent hooks and pipeline stages.
The context is built up in the following order: base context (request, session, helpers, ...) ->
extendContext -> preNavigationHooks -> navigation -> postNavigationHooks -> requestHandler.
This means the members added by extendContext are already available here, but navigation-dependent
members (e.g. response, body, $) are not.
Example:
preNavigationHooks: [
async (crawlingContext) => {
// ...
},
]
optionalinheritedproxyConfiguration
If set, the crawler will be configured for all connections to use the Proxy URLs provided and rotated according to the configuration.
optionalinheritedrequestHandler
User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
The function receives the BasicCrawlingContext as an argument,
where the request represents the URL to crawl.
The function must return a promise, which is then awaited by the crawler.
If the function throws an exception, the crawler will try to re-crawl the
request later, up to the maxRequestRetries times.
If all the retries fail, the crawler calls the function
provided to the failedRequestHandler parameter.
To make this work, we should always
let our function throw exceptions rather than catch them.
The exceptions are logged to the request using the
Request.pushErrorMessage() function.
optionalinheritedrequestHandlerTimeoutSecs
Timeout in which the function passed as requestHandler needs to finish, in seconds.
optionalinheritedrequestList
Static list of URLs to be processed.
Deprecated - Use the requestManager option instead. To combine a read-only loader (such as a RequestList)
with a writable queue, build a tandem with requestList.toTandem(requestQueue)
and pass the result as requestManager. When both requestList and requestQueue are provided, they are
combined into a tandem automatically.
optionalinheritedrequestManager
Manager of requests that should be processed by the crawler. Mutually exclusive with the deprecated
requestQueue and requestList options.
If not provided, the crawler will open the default RequestQueue when it is first needed.
optionalinheritedrequestQueue
Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
Deprecated - Use the requestManager option instead. A RequestQueue is itself a request manager, so you can
pass it directly as requestManager.
optionalinheritedrespectRobotsTxtFile
If set to true, the crawler will automatically try to fetch the robots.txt file for each domain,
and skip those that are not allowed. This also prevents disallowed URLs to be added via enqueueLinks.
If an object is provided, it may contain a userAgent property to specify which user-agent
should be used when checking the robots.txt file. If not provided, the default user-agent * will be used.
optionalinheritedretryOnBlocked
If set to true, the crawler will automatically try to bypass any detected bot protection.
Currently supports:
optionalinheritedsameDomainDelaySecs
Indicates how much time (in seconds) to wait before crawling another same domain request. Subdomains are paced together with the site they belong to.
Wraps the crawler's request manager in a ThrottlingRequestManager; pass one as requestManager
yourself to configure it further.
optionalinheritedsaveResponseCookies
Automatically saves cookies to Session. Enabled by default.
It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request. It passes the "Cookie" header to the request with the session cookies.
optionalinheritedsessionPool
An existing session pool instance to use. When provided, the crawler will use this pool directly instead of creating a new one, enabling session sharing across multiple crawlers. The crawler will not tear down a shared pool — the caller is responsible for its lifecycle.
Accepts the built-in SessionPool or any object implementing the ISessionPool interface, so custom session-management strategies can be plugged in.
optionalinheritedstatistics
A preconfigured statistics instance. When provided, the crawler records into it instead of building its own and
will not reset() it between run() calls. Accepts the built-in Statistics or any object
implementing IStatistics.
Custom fields declared via stateExtension are carried over to
crawler.statistics.state:
const statistics = new Statistics({ stateExtension: { defaultState: { productsFound: 0 } } });
const crawler = new BasicCrawler({
statistics,
requestHandler: async () => {
statistics.state.productsFound++;
},
});
await crawler.run();
// the custom fields are typed on `crawler.statistics` too
console.log(crawler.statistics.state.productsFound);
optionalinheritedstatusMessageCallback
Allows overriding the default status message. The callback needs to call crawler.setStatusMessage() explicitly.
The default status message is provided in the parameters.
const crawler = new CheerioCrawler({
statusMessageCallback: async (ctx) => {
return ctx.crawler.setStatusMessage(`this is status message from ${new Date().toISOString()}`, { level: 'INFO' }); // log level defaults to 'DEBUG'
},
statusMessageLoggingInterval: 1, // defaults to 10s
async requestHandler({ $, enqueueLinks, request, log }) {
// ...
},
});
optionalinheritedstatusMessageLoggingInterval
Defines the length of the interval for calling the setStatusMessage in seconds.
optionalinheritedstorageBackend
Custom storage backend to use for this crawler. If provided, the crawler will use its own ServiceLocator instance instead of the global one.
optionalinheritedsuggestResponseEncoding
By default this crawler will extract correct encoding from the HTTP response headers.
Sadly, there are some websites which use invalid headers. Those are encoded using the UTF-8 encoding.
If those sites actually use a different encoding, the response will be corrupted. You can use
suggestResponseEncoding to fall back to a certain encoding, if you know that your target website uses it.
To force a certain encoding, disregarding the response headers, use HttpCrawlerOptions.forceResponseEncoding
// Will fall back to windows-1250 encoding if none found
suggestResponseEncoding: 'windows-1250'
optionalinheritedtaskLoopOptions
Lets you override the predicates that steer the crawler's task loop: isTaskReadyFunction (may another request
start?) and isFinishedFunction (is the crawl over?). The task itself — fetching a request and running it
through the pipeline — is owned by the crawler and cannot be overridden.
Concurrency is configured elsewhere — through the minConcurrency/maxConcurrency/maxRequestsPerMinute
shortcuts, or a concurrencySystem for finer control.
optionalinheritedtransactionalStorage
Makes the storage writes performed while handling a request atomic with respect to the request succeeding: they are recorded in a StorageTransaction spanning the whole request lifecycle and only applied when the request handler succeeds, so a thrown handler leaves no partial writes behind and a retry does not double-write. Reads within the handler see its own writes.
false disables the mechanism entirely; an object overrides the per-storage-type
StorageWritePolicy (e.g. { requestQueue: 'deferred' } for all-or-nothing enqueues).
withDirectStorageAccess is the per-call-site escape hatch; useState() is deliberately
not transactional.
An array of additional HTTP response Status Codes to be treated as errors. By default, status codes >= 500 trigger errors.