Skip to main content
Version: Next

@crawlee/basic

Provides a simple framework for parallel crawling of web pages. The URLs to crawl are fed either from a static list of URLs or from a dynamic queue of URLs enabling recursive crawling of websites.

BasicCrawler is a low-level tool that requires the user to implement the page download and data extraction functionality themselves. If we want a crawler that already facilitates this functionality, we should consider using CheerioCrawler, PuppeteerCrawler or PlaywrightCrawler.

BasicCrawler invokes the user-provided requestHandler for each Request object, which represents a single URL to crawl. The Request objects are fed from the RequestList or RequestQueue instances provided by the requestList or requestQueue constructor options, respectively. If neither requestList nor requestQueue options are provided, the crawler will open the default request queue either when the crawler.addRequests() function is called, or if requests parameter (representing the initial requests) of the crawler.run() function is provided.

If both requestList and requestQueue options are used, the instance first processes URLs from the RequestList and automatically enqueues all of them to the RequestQueue before it starts their processing. This ensures that a single URL is not crawled multiple times.

The crawler finishes if there are no more Request objects to crawl.

New requests are only dispatched when there is enough free CPU and memory available, as judged by the crawler's ConcurrencySystem. Concurrency is tuned via the minConcurrency, maxConcurrency and maxRequestsPerMinute options of the BasicCrawler constructor, or, for finer control, by injecting a pre-configured concurrencySystem.

Example usage​

import { BasicCrawler, Dataset } from 'crawlee';

// Create a crawler instance
const crawler = new BasicCrawler({
    async requestHandler({ request, sendRequest }) {
        // 'request' contains an instance of the Request class
        // Here we simply fetch the HTML of the page and store it to a dataset
        const { body } = await sendRequest({
            url: request.url,
            method: request.method,
            body: request.payload,
            headers: request.headers,
        });

        await Dataset.pushData({
            url: request.url,
            html: body,
        })
    },
});

// Enqueue the initial requests and run the crawler
await crawler.run([
    'http://www.example.com/page-1',
    'http://www.example.com/page-2',
]);

Index

Crawlers

Scaling

Sources

Other

Other

AddRequestsBatchedOptions

Re-exports AddRequestsBatchedOptions

AddRequestsBatchedResult

Re-exports AddRequestsBatchedResult

AfterCommitError

Re-exports AfterCommitError

ApifyLogAdapter

Re-exports ApifyLogAdapter

ArgumentValidationError

Re-exports ArgumentValidationError

BaseCrawleeLogger

Re-exports BaseCrawleeLogger

coerceBoolean

Re-exports coerceBoolean

coerceNumber

Re-exports coerceNumber

ConfigField

Re-exports ConfigField

Configuration

Re-exports Configuration

ConfigurationInput

Re-exports ConfigurationInput

ConfigurationOptions

Re-exports ConfigurationOptions

crawleeConfigFields

Re-exports crawleeConfigFields

CrawleeLogger

Re-exports CrawleeLogger

CrawleeLoggerOptions

Re-exports CrawleeLoggerOptions

createStorageTransaction

Re-exports createStorageTransaction

CriticalError

Re-exports CriticalError

Dataset

Re-exports Dataset

DatasetConsumer

Re-exports DatasetConsumer

DatasetContent

Re-exports DatasetContent

DatasetDataOptions

Re-exports DatasetDataOptions

DatasetExportOptions

Re-exports DatasetExportOptions

DatasetExportToOptions

Re-exports DatasetExportToOptions

DatasetIteratorOptions

Re-exports DatasetIteratorOptions

DatasetMapper

Re-exports DatasetMapper

DatasetReducer

Re-exports DatasetReducer

DatasetStats

Re-exports DatasetStats

DefaultStorageIdentifier

Re-exports DefaultStorageIdentifier

EnqueueStrategy

Re-exports EnqueueStrategy

EnqueueStrategyOption

Re-exports EnqueueStrategyOption

EventManager

Re-exports EventManager

EventManagerOptions

Re-exports EventManagerOptions

EventStatusMessageData

Re-exports EventStatusMessageData

EventType

Re-exports EventType

EventTypeName

Re-exports EventTypeName

ExplicitStorageIdentifier

Re-exports ExplicitStorageIdentifier

field

Re-exports field

FieldsInput

Re-exports FieldsInput

FieldsOutput

Re-exports FieldsOutput

IProxyConfiguration

Re-exports IProxyConfiguration

IRequestLoader

Re-exports IRequestLoader

IRequestManager

Re-exports IRequestManager

IStorage

Re-exports IStorage

KeyConsumer

Re-exports KeyConsumer

KeyValueStore

Re-exports KeyValueStore

KeyValueStoreIteratorOptions

Re-exports KeyValueStoreIteratorOptions

KeyValueStoreRawRecord

Re-exports KeyValueStoreRawRecord

KeyValueStoreStats

Re-exports KeyValueStoreStats

LoadSignalInfo

Re-exports LoadSignalInfo

LocalEventManager

Re-exports LocalEventManager

LocalEventManagerOptions

Re-exports LocalEventManagerOptions

log

Re-exports log

Log

Re-exports Log

Logger

Re-exports Logger

LoggerJson

Re-exports LoggerJson

LoggerOptions

Re-exports LoggerOptions

LoggerText

Re-exports LoggerText

LogLevel

Re-exports LogLevel

LogOptions

Re-exports LogOptions

MemoryStorageBackend

Re-exports MemoryStorageBackend

MemoryStorageOptions

Re-exports MemoryStorageOptions

NonRetryableError

Re-exports NonRetryableError

PacingScope

Re-exports PacingScope

PacingSignal

Re-exports PacingSignal

parseValue

Re-exports parseValue

ProxyConfiguration

Re-exports ProxyConfiguration

ProxyConfigurationFunction

Re-exports ProxyConfigurationFunction

ProxyConfigurationOptions

Re-exports ProxyConfigurationOptions

purgeDefaultStorages

Re-exports purgeDefaultStorages

RecordOptions

Re-exports RecordOptions

RecoverableState

Re-exports RecoverableState

RecoverableStateBaseOptions

Re-exports RecoverableStateBaseOptions

RecoverableStateOptions

Re-exports RecoverableStateOptions

RecoverableStatePersistenceOptions

Re-exports RecoverableStatePersistenceOptions

Request

Re-exports Request

RequestList

Re-exports RequestList

RequestListOptions

Re-exports RequestListOptions

RequestListSource

Re-exports RequestListSource

RequestListSourcesFunction

Re-exports RequestListSourcesFunction

RequestListState

Re-exports RequestListState

RequestLoaderStatus

Re-exports RequestLoaderStatus

RequestManagerTandem

Re-exports RequestManagerTandem

RequestOptions

Re-exports RequestOptions

RequestQueue

Re-exports RequestQueue

RequestQueueOperationInfo

Re-exports RequestQueueOperationInfo

RequestQueueOperationOptions

Re-exports RequestQueueOperationOptions

RequestQueueStats

Re-exports RequestQueueStats

RequestsLike

Re-exports RequestsLike

RequestSourceStatus

Re-exports RequestSourceStatus

RequestValidationError

Re-exports RequestValidationError

ResolvedConfigValues

Re-exports ResolvedConfigValues

SchemaIssue

Re-exports SchemaIssue

serializeValue

Re-exports serializeValue

ServiceConflictError

Re-exports ServiceConflictError

serviceLocator

Re-exports serviceLocator

ServiceLocator

Re-exports ServiceLocator

SessionError

Re-exports SessionError

SessionRetiredError

Re-exports SessionRetiredError

Source

Re-exports Source

StateConversion

Re-exports StateConversion

StateValidationError

Re-exports StateValidationError

StorageBackend

Re-exports StorageBackend

StorageIdentifier

Re-exports StorageIdentifier

StorageInstanceManager

Re-exports StorageInstanceManager

StorageOpenOptions

Re-exports StorageOpenOptions

StorageTransaction

Re-exports StorageTransaction

StorageTransactionOptions

Re-exports StorageTransactionOptions

StorageTransactionState

Re-exports StorageTransactionState

StorageTransactionView

Re-exports StorageTransactionView

StorageWriteMode

Re-exports StorageWriteMode

StorageWritePolicy

Re-exports StorageWritePolicy

SyncStateConversion

Re-exports SyncStateConversion

SystemInfo

Re-exports SystemInfo

useState

Re-exports useState

UseStateOptions

Re-exports UseStateOptions

withDirectStorageAccess

Re-exports withDirectStorageAccess

withStorageTransaction

Re-exports withStorageTransaction

CleanupRegistrar

CleanupRegistrar: (cleanup) => void

Registers a callback that runs after the final context consumer finishes. Cleanups run in reverse registration order and receive the same error as the onError parameter of ContextPipeline.call: undefined on success, otherwise a ContextPipelineInitializationError (a middleware failed) or a RequestHandlerError (the consumer failed), with the original in cause. A cleanup that throws fails the whole call with a ContextPipelineCleanupError, which is critical.

Type declaration

    • (cleanup): void

    • Parameters

      Returns void

ContextMiddleware

ContextMiddleware<TCrawlingContext, TCrawlingContextExtension>: (context, onCleanup) => Awaitable<TCrawlingContextExtension>

A middleware step in the context pipeline: receives the context built so far and returns an extension that gets merged into it. Anything that needs tearing down is registered via onCleanup.

Type parameters

  • TCrawlingContext

    The input context type for this middleware

  • TCrawlingContextExtension

    The enhanced output context type

Type declaration

DefaultRouteUserData

DefaultRouteUserData<Routes, Fallback>: Routes extends { [defaultRoute]: infer DefaultUserData } ? DefaultUserData : Fallback

The userData type of the default route: inferred from the defaultRoute schema when the route map carries one, otherwise the provided Fallback.

Type parameters

EnqueueLinksOptions

EnqueueLinksOptions: ExtractLinksOptions & EnqueueUrlsOptions

The combined options accepted by a crawler context's enqueueLinks() helper: extractLinks() + enqueueUrls().

ErrorHandler

ErrorHandler<BaseContext, ExtendedContext>: (inputs, error) => Awaitable<void>

An error handler receives the crawling context and the error that was thrown while processing the request.

Unlike the RequestHandler, an error handler may run before the context pipeline has finished building the full context (e.g. when navigation or session setup fails). Therefore only BaseContext is guaranteed to be present, while the extra properties added by the pipeline and extendContext (the difference between BaseContext and ExtendedContext) are only available as a Partial.

Type parameters

Type declaration

    • Parameters

      • inputs: BaseContext & Partial<ExtendedContext>
      • error: Error

      Returns Awaitable<void>

GetUserDataFromRequest

GetUserDataFromRequest<T>: T extends CrawlingRequest<infer Y> ? Y : never

Type parameters

  • T

GlobInput

GlobInput: string | GlobObject

LabeledSource

LabeledSource<Routes>: string extends keyof Routes ? string | Source : string | Request | ({ regex?: RegExp; requestsFromUrl?: string } & ({ [ Label in keyof Routes & string ]: Omit<Partial<...>, label> & { label: Label } }[keyof Routes & string] | (Omit<Partial<RequestOptions>, label> & { label?: undefined })))

A request input (URL string, request-options object, or Request) whose userData is typed according to its label, based on a router's route map.

When the route map is open (the default Record<string, ...>), this is just the regular loose Source input. When the map declares concrete labels, providing a label requires the matching userData shape and rejects labels not present in the map; unlabeled requests keep loose userData.

Type parameters

LoadedRequest

LoadedRequest<R>: R & Required<Pick<R, id | loadedUrl>>

A Request that has been dispatched, so its id and loadedUrl are guaranteed to be present.

Type parameters

RegExpInput

RegExpInput: RegExp | RegExpObject

RequestHandler

RequestHandler<Context>: (inputs) => Awaitable<void>

Type parameters

Type declaration

RequireContextPipeline

RequireContextPipeline<DefaultContextType, FinalContextType>: DefaultContextType extends FinalContextType ? {} : { contextPipelineBuilder: () => ContextPipeline<CrawlingContext, FinalContextType> }

Type parameters

RouterHandlerContext

RouterHandlerContext<Context, UserData, Routes>: Omit<Context, request | addRequests | enqueueLinks> & { addRequests: TypedContextAddRequests<Routes>; request: LoadedRequest<CrawlingRequest<UserData>> } & (Context extends { enqueueLinks: infer EnqueueLinks } ? { enqueueLinks: TypedContextEnqueueLinks<EnqueueLinks, Routes> } : {})

The crawling context received by a route handler, with request.userData narrowed to UserData, and addRequests/enqueueLinks typed according to the router's route map (Routes) so that enqueuing a request under a declared label requires the matching userData shape.

Type parameters

RouterLabel

RouterLabel<Routes>: string extends keyof Routes ? string | symbol : (keyof Routes & string) | symbol

The set of labels accepted by Router.addHandler. When the router declares a concrete route map (e.g. { PRODUCT: ...; CATEGORY: ... }), only those labels (plus symbols) are allowed — unknown labels become a compile-time error. When the map is left open (the default Record<string, ...>), any string or symbol label is accepted, preserving the original behaviour.

Type parameters

RouterRoutes

RouterRoutes<Context, Routes>: { [ Label in keyof Routes ]: (ctx) => Awaitable<void> }

Type parameters

RouteSchemas

RouteSchemas: Record<string, StandardSchemaV1> & { [defaultRoute]?: StandardSchemaV1 }

A map of request labels to a Standard Schema (Zod, Valibot, ArkType, …) validating that label's request.userData. Pass it to Router.create or a createXRouter factory to derive the per-label request.userData types and validate them at runtime. The optional defaultRoute key registers a schema for requests handled by the default route.

RoutesFromSchemas

RoutesFromSchemas<Schemas>: { [ Label in Extract<keyof Schemas, string> ]: SchemaUserData<Schemas[Label]> } & (Schemas extends { [defaultRoute]: StandardSchemaV1 } ? { [defaultRoute]: SchemaUserData<Schemas[typeof defaultRoute]> } : {})

Derives a route map (label → userData type) from a RouteSchemas map by inferring each schema's output type. Outputs that are not object-shaped fall back to a plain Dictionary. The defaultRoute schema is kept under its symbol key so Router.addDefaultHandler can pick it up; string labels (the ones Router.addHandler and the crawler-level typing accept) ignore it.

Type parameters

SessionReuseStrategy

SessionReuseStrategy: random | round-robin | use-until-failure

SkippedRequestCallback

SkippedRequestCallback: (args) => Awaitable<void>

Type declaration

SkippedRequestReason

SkippedRequestReason: robotsTxt | limit | enqueueLimit | filters | transform | redirect | depth

StatusMessageCallback

StatusMessageCallback<Context, Crawler>: (params) => Awaitable<void>

Type parameters

Type declaration

TypedContextAddRequests

TypedContextAddRequests<Routes>: (requestsLike, options) => Promise<AddRequestsBatchedResult>

The label-aware addRequests method signature exposed on a request handler's context when the crawler is bound to a typed router. Mirrors RestrictedCrawlingContext.addRequests with typed sources.

Type parameters

Type declaration

TypedContextEnqueueLinks

TypedContextEnqueueLinks<EnqueueLinks, Routes>: EnqueueLinks extends (options) => infer Result ? (options) => Result : EnqueueLinks extends (options) => infer Result ? (options) => Result : EnqueueLinks

Transforms a context's existing enqueueLinks method so that the label/userData in its options follow the router's route map, while preserving everything else about the signature (argument optionality and return type, which differ between crawler types).

Type parameters

  • EnqueueLinks
  • Routes: Record<keyof Routes, Dictionary>

TypedRequestsLike

TypedRequestsLike<Routes>: AsyncIterable<LabeledSource<Routes>> | Iterable<LabeledSource<Routes>> | LabeledSource<Routes>[]

The iterable/array of LabeledSource inputs accepted by the label-aware addRequests/run methods of a crawler bound to a typed router.

Type parameters

UrlPatternInput

UrlPatternInput: GlobInput | RegExpInput

Unified URL pattern input — accepts glob strings, glob objects, RegExp instances, or regexp objects.

constBLOCKED_STATUS_CODES

BLOCKED_STATUS_CODES: readonly number[] = ...

constdefaultRoute

defaultRoute: unique symbol = ...

The key of the default route — the fallback handler registered via Router.addDefaultHandler. Use it in a RouteSchemas map to register a schema that validates the userData of every request that falls through to the default handler (i.e. whose label has no route of its own).

Page Options