Skip to main content
Version: Next

@crawlee/core

Core set of classes required for Crawlee.

The crawlee package consists of several smaller packages, released separately under @crawlee namespace:

Installing Crawlee​

Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. @crawlee/playwright if you plan on using playwright - it already contains everything from the @crawlee/browser package, which includes everything from @crawlee/basic, which includes everything from @crawlee/core.

If we don't care much about additional code being pulled in, we can just use the crawlee meta-package, which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

npm install crawlee

Or if all we need is cheerio support, we can install only @crawlee/cheerio.

npm install @crawlee/cheerio

When using playwright or puppeteer, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.

npm install crawlee playwright
# or npm install @crawlee/playwright playwright

Alternatively we can also use the crawlee meta-package which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

Sometimes you might want to use some utility methods from @crawlee/utils, so you might want to install that as well. This package contains some utilities that were previously available under Apify.utils. Browser related utilities can be also found in the crawler packages (e.g. @crawlee/playwright).

Index

Crawlers

Result Stores

Scaling

Sources

Other

Other

ConfigurationInput

ConfigurationInput: FieldsInput<typeof crawleeConfigFields>

ConfigurationOptions

ConfigurationOptions: ConfigurationInput

Deprecated - Use ConfigurationInput instead.

DefaultRouteUserData

DefaultRouteUserData<Routes, Fallback>: Routes extends { [defaultRoute]: infer DefaultUserData } ? DefaultUserData : Fallback

The userData type of the default route: inferred from the defaultRoute schema when the route map carries one, otherwise the provided Fallback.

Type parameters

EnqueueLinksOptions

EnqueueLinksOptions: ExtractLinksOptions & EnqueueUrlsOptions

The combined options accepted by a crawler context's enqueueLinks() helper: extractLinks() + enqueueUrls().

EnqueueStrategyOption

EnqueueStrategyOption: EnqueueStrategy | all | same-domain | same-hostname | same-origin

The strategy option accepted by ExtractLinksOptions and EnqueueUrlsOptions.

EventTypeName

EventTypeName: EventType | systemInfo | persistState | migrating | aborting | exit | statusMessage

ExplicitStorageIdentifier

ExplicitStorageIdentifier: { alias?: never; id: string; name?: never } | { alias?: never; id?: never; name: string } | { alias: string; id?: never; name?: never }

A storage identifier where exactly one of id, name, or alias is specified. Produced by resolveStorageIdentifier from ambiguous user input.

FieldsInput

FieldsInput<F>: { [ K in keyof F ]?: z.output<F[K][schema]> }

Type parameters

FieldsOutput

FieldsOutput<F>: { [ K in keyof F ]: z.output<F[K][schema]> }

Type parameters

GetUserDataFromRequest

GetUserDataFromRequest<T>: T extends Request<infer Y> ? Y : never

Type parameters

  • T

GlobInput

GlobInput: string | GlobObject

JournalEntry

JournalEntry: DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry

LabeledSource

LabeledSource<Routes>: string extends keyof Routes ? string | Source : string | Request | ({ regex?: RegExp; requestsFromUrl?: string } & ({ [ Label in keyof Routes & string ]: Omit<Partial<...>, label> & { label: Label } }[keyof Routes & string] | (Omit<Partial<RequestOptions>, label> & { label?: undefined })))

A request input (URL string, request-options object, or Request) whose userData is typed according to its label, based on a router's route map.

When the route map is open (the default Record<string, ...>), this is just the regular loose Source input. When the map declares concrete labels, providing a label requires the matching userData shape and rejects labels not present in the map; unlabeled requests keep loose userData.

Type parameters

LoadedRequest

LoadedRequest<R>: WithRequired<R, id | loadedUrl>

Type parameters

PacingScope

PacingScope: LiteralUnion<hostname | registrableDomain, string>

How much of the URL space a PacingSignal covers.

Open on purpose: 'hostname' and 'registrableDomain' are what Crawlee's own reporters send and what ThrottlingRequestManager understands, but any string is accepted, so a pacer keyed on something else can be reported to in its own vocabulary.

PacingSignal

PacingSignal: { reason: rateLimited; scope?: PacingScope; url: string; waitMs?: number } | { intervalMs: number; reason: minInterval; scope: PacingScope; url: string } | { intervalMs: number; reason: minIntervalEverywhere; scope: PacingScope }

Something said about the pace requests should go out at, reported to a request manager through IRequestManager.recordPacingSignal.

One shape rather than a method per channel: a pacing manager switches on reason, a wrapping one forwards the value without knowing what is in it, and a new kind of signal costs the interface nothing. The url travels inside the value because the crawl-wide variant has none. Nothing here names the mechanism a signal came from - status codes, headers and robots.txt are the crawler's business - and every delay is in milliseconds.

Scope

A manager may apply a signal to a wider scope than it was given - a floor that holds for one host still holds when a whole site is paced by it - but never to a narrower one, which would leave some of the URLs the signal covers running unpaced. A manager that can only do the latter, or that does not recognise the scope at all, MUST throw rather than quietly under-apply it.

RegExpInput

RegExpInput: RegExp | RegExpObject

RequestListSourcesFunction

RequestListSourcesFunction: () => Promise<RequestListSource[]>

Type declaration

    • (): Promise<RequestListSource[]>

    • Returns Promise<RequestListSource[]>

RequestLoaderStatus

RequestLoaderStatus: Exclude<RequestSourceStatus, { status: stalled }>

Loaders never stall — only a manager that paces its own dispatch can.

RequestManagerOpener

RequestManagerOpener<T>: (identifier, options) => Promise<T>

Opens a request manager, matching the shape of storage open methods such as RequestQueue.open.

ThrottlingRequestManager calls this once per configured domain, so every per-domain queue shares the concrete type and storage backend of the manager being wrapped.

Type parameters

Type declaration

RequestsLike

RequestsLike: AsyncIterable<Source | string> | Iterable<Source | string> | (Source | string)[]

RequestSourceStatus

RequestSourceStatus: { status: ready } | { readyAt?: number; status: waiting } | { reason: string; status: stalled } | { status: finished }

A request source's own availability, in a single answer.

  • ready — the next IRequestLoader.fetchNextRequest is expected to hand something over.
  • waiting — nothing to fetch right now, but the source is not done: requests are in progress, are being added in the background, or are held back until readyAt.
  • stalled — the source holds requests it cannot make progress on. Only a manager that paces its own dispatch can reach this; see ThrottlingRequestManager.
  • finished — everything has been handled.

ResolvedConfigValues

ResolvedConfigValues: FieldsOutput<typeof crawleeConfigFields>

RouterHandlerContext

RouterHandlerContext<Context, UserData, Routes>: Omit<Context, request | addRequests | enqueueLinks> & { addRequests: TypedContextAddRequests<Routes>; request: LoadedRequest<Request<UserData>> } & (Context extends { enqueueLinks: infer EnqueueLinks } ? { enqueueLinks: TypedContextEnqueueLinks<EnqueueLinks, Routes> } : {})

The crawling context received by a route handler, with request.userData narrowed to UserData, and addRequests/enqueueLinks typed according to the router's route map (Routes) so that enqueuing a request under a declared label requires the matching userData shape.

Type parameters

RouterLabel

RouterLabel<Routes>: string extends keyof Routes ? string | symbol : (keyof Routes & string) | symbol

The set of labels accepted by Router.addHandler. When the router declares a concrete route map (e.g. { PRODUCT: ...; CATEGORY: ... }), only those labels (plus symbols) are allowed — unknown labels become a compile-time error. When the map is left open (the default Record<string, ...>), any string or symbol label is accepted, preserving the original behaviour.

Type parameters

RouterRoutes

RouterRoutes<Context, Routes>: { [ Label in keyof Routes ]: (ctx) => Awaitable<void> }

Type parameters

RouteSchemas

RouteSchemas: Record<string, StandardSchemaV1> & { [defaultRoute]?: StandardSchemaV1 }

A map of request labels to a Standard Schema (Zod, Valibot, ArkType, …) validating that label's request.userData. Pass it to Router.create or a createXRouter factory to derive the per-label request.userData types and validate them at runtime. The optional defaultRoute key registers a schema for requests handled by the default route.

RoutesFromSchemas

RoutesFromSchemas<Schemas>: { [ Label in Extract<keyof Schemas, string> ]: SchemaUserData<Schemas[Label]> } & (Schemas extends { [defaultRoute]: StandardSchemaV1 } ? { [defaultRoute]: SchemaUserData<Schemas[typeof defaultRoute]> } : {})

Derives a route map (label → userData type) from a RouteSchemas map by inferring each schema's output type. Outputs that are not object-shaped fall back to a plain Dictionary. The defaultRoute schema is kept under its symbol key so Router.addDefaultHandler can pick it up; string labels (the ones Router.addHandler and the crawler-level typing accept) ignore it.

Type parameters

SessionReuseStrategy

SessionReuseStrategy: typeof SESSION_REUSE_STRATEGIES[number]

SkippedRequestCallback

SkippedRequestCallback: (args) => Awaitable<void>

Type declaration

SkippedRequestReason

SkippedRequestReason: robotsTxt | limit | enqueueLimit | filters | transform | redirect | depth

Source

Source: (Partial<RequestOptions> & { regex?: RegExp; requestsFromUrl?: string }) | CrawleeRequest

StateConversion

StateConversion<TFrom, TTo>: (value) => Awaitable<TTo> | StandardSchemaV1<TFrom, TTo>

One direction of the conversion between the state model and its persisted form - either a plain function, or a Standard Schema whose validated output is the result.

A schema that fails to validate makes RecoverableState throw a StateValidationError. Zod codecs work directly, as their validation is the decode direction; use (state) => codec.encode(state) for the other one.

Type parameters

  • TFrom
  • TTo

StorageIdentifier

StorageIdentifier: { alias?: never; id: string; name?: never } | { alias?: never; id?: never; name: string } | { alias: string; id?: never; name?: never } | { alias?: never; id?: never; name?: never }

Identifies a storage by its ID, name, or alias. At most one may be provided.

  • { id } — open a pre-existing storage by its unique ID.
  • { name } — open or create a globally named storage (persists across runs). The name default is reserved: it resolves to the default storage, and is emptied on start along with it.
  • { alias } — open or create a run-scoped unnamed storage identified by this alias. The alias is used locally (e.g. as a directory name or cache key) but the storage itself has no persistent name. Use this for non-default unnamed storages. Like the default storage, an aliased one is emptied on start unless purgeOnStart is disabled.
  • {} / omitted — open the default storage.

StorageTransactionState

StorageTransactionState: open | committing | committed | failed | rolledBack

StorageWriteMode

StorageWriteMode: deferred | writeThrough

Governs whether writes of a given storage type performed inside a StorageTransaction are applied immediately (writeThrough) or recorded and replayed on commit (deferred).

SyncStateConversion

SyncStateConversion<TFrom, TTo>: (value) => TTo | StandardSchemaV1<TFrom, TTo>

A StateConversion for a caller that cannot await one - Statistics, whose toJSON() is synchronous, being the reason this exists.

Only the function arm can be narrowed here: a Standard Schema is free to validate asynchronously, so a schema that does is rejected when it runs rather than when it is passed.

Type parameters

  • TFrom
  • TTo

TypedContextAddRequests

TypedContextAddRequests<Routes>: (requestsLike, options) => Promise<AddRequestsBatchedResult>

The label-aware addRequests method signature exposed on a request handler's context when the crawler is bound to a typed router. Mirrors RestrictedCrawlingContext.addRequests with typed sources.

Type parameters

Type declaration

TypedContextEnqueueLinks

TypedContextEnqueueLinks<EnqueueLinks, Routes>: EnqueueLinks extends (options) => infer Result ? (options) => Result : EnqueueLinks extends (options) => infer Result ? (options) => Result : EnqueueLinks

Transforms a context's existing enqueueLinks method so that the label/userData in its options follow the router's route map, while preserving everything else about the signature (argument optionality and return type, which differ between crawler types).

Type parameters

  • EnqueueLinks
  • Routes: Record<keyof Routes, Dictionary>

UrlPatternInput

UrlPatternInput: GlobInput | RegExpInput

Unified URL pattern input — accepts glob strings, glob objects, RegExp instances, or regexp objects.

WithRequired

WithRequired<T, K>: T & { [ P in K ]-?: T[P] }

Type parameters

  • T
  • K: keyof T

constBLOCKED_STATUS_CODES

BLOCKED_STATUS_CODES: number[] = ...

constcoerceBoolean

coerceBoolean: ZodPreprocess<ZodBoolean> = ...

Zod preprocessor treating '0' and 'false' as falsy.

constcoerceNumber

coerceNumber: ZodPreprocess<ZodNumber> = ...

constcrawleeConfigFields

crawleeConfigFields: { availableMemoryRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; chromeExecutablePath: ConfigField<ZodOptional<ZodString>>; containerized: ConfigField<ZodOptional<ZodPreprocess<ZodBoolean>>>; defaultBrowserPath: ConfigField<ZodOptional<ZodString>>; defaultDatasetId: ConfigField<ZodDefault<ZodString>>; defaultKeyValueStoreId: ConfigField<ZodDefault<ZodString>>; defaultRequestQueueId: ConfigField<ZodDefault<ZodString>>; disableBrowserSandbox: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; headless: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; inputKey: ConfigField<ZodDefault<ZodString>>; internalTimeoutMillis: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>; logLevel: ConfigField<ZodOptional<ZodPreprocess<ZodEnum<typeof LogLevel>>>>; maxUsedCpuRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; memoryMbytes: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>; persistStateIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; persistStorage: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; purgeOnStart: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; storageDir: ConfigField<ZodDefault<ZodString>>; systemInfoIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; xvfb: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>> } = ...

Type declaration

  • availableMemoryRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>
  • chromeExecutablePath: ConfigField<ZodOptional<ZodString>>
  • containerized: ConfigField<ZodOptional<ZodPreprocess<ZodBoolean>>>
  • defaultBrowserPath: ConfigField<ZodOptional<ZodString>>
  • defaultDatasetId: ConfigField<ZodDefault<ZodString>>
  • defaultKeyValueStoreId: ConfigField<ZodDefault<ZodString>>
  • defaultRequestQueueId: ConfigField<ZodDefault<ZodString>>
  • disableBrowserSandbox: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>
  • headless: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>
  • inputKey: ConfigField<ZodDefault<ZodString>>
  • internalTimeoutMillis: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>

    Internal safety-net timeout for a single request, in milliseconds. When unset the crawler derives it from the request handler timeout (twice it, and never below 5 minutes).

  • logLevel: ConfigField<ZodOptional<ZodPreprocess<ZodEnum<typeof LogLevel>>>>
  • maxUsedCpuRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>
  • memoryMbytes: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>
  • persistStateIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>
  • persistStorage: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>
  • purgeOnStart: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>
  • storageDir: ConfigField<ZodDefault<ZodString>>
  • systemInfoIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>
  • xvfb: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>

constdefaultRoute

defaultRoute: unique symbol = ...

The key of the default route — the fallback handler registered via Router.addDefaultHandler. Use it in a RouteSchemas map to register a schema that validates the userData of every request that falls through to the default handler (i.e. whose label has no route of its own).

externalconstlog

log: Log

constMAX_POOL_SIZE

MAX_POOL_SIZE: 1000 = 1000

constPERSIST_STATE_KEY

PERSIST_STATE_KEY: CRAWLEE_SESSION_POOL_STATE = 'CRAWLEE_SESSION_POOL_STATE'

constserviceLocator

serviceLocator: ServiceLocatorInterface = ...
Page Options