Skip to main content
Version: Next

@crawlee/core

Core set of classes required for Crawlee.

The crawlee package consists of several smaller packages, released separately under @crawlee namespace:

Installing Crawlee​

Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. @crawlee/playwright if you plan on using playwright - it already contains everything from the @crawlee/browser package, which includes everything from @crawlee/basic, which includes everything from @crawlee/core.

If we don't care much about additional code being pulled in, we can just use the crawlee meta-package, which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

npm install crawlee

Or if all we need is cheerio support, we can install only @crawlee/cheerio.

npm install @crawlee/cheerio

When using playwright or puppeteer, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.

npm install crawlee playwright
# or npm install @crawlee/playwright playwright

Alternatively we can also use the crawlee meta-package which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

Sometimes you might want to use some utility methods from @crawlee/utils, so you might want to install that as well. This package contains some utilities that were previously available under Apify.utils. Browser related utilities can be also found in the crawler packages (e.g. @crawlee/playwright).

Index

Crawlers

Result Stores

Scaling

Sources

Other

Other

RequestQueueV2

Renames and re-exports RequestQueue

EventTypeName

EventTypeName: EventType | systemInfo | persistState | migrating | aborting | exit

GetUserDataFromRequest

GetUserDataFromRequest<T>: T extends Request<infer Y> ? Y : never

Type parameters

  • T

GlobInput

GlobInput: string | GlobObject

GlobObject

GlobObject: { glob: string } & Pick<RequestOptions, method | payload | label | userData | headers>

LoadedRequest

LoadedRequest<R>: WithRequired<R, id | loadedUrl>

Type parameters

PseudoUrlInput

PseudoUrlInput: string | PseudoUrlObject

PseudoUrlObject

PseudoUrlObject: { purl: string } & Pick<RequestOptions, method | payload | label | userData | headers>

RedirectHandler

RedirectHandler: (redirectResponse, updatedRequest) => void

Type of a function called when an HTTP redirect takes place. It is allowed to mutate the updatedRequest argument.

Type declaration

    • (redirectResponse, updatedRequest): void

    • Parameters

      • redirectResponse: BaseHttpResponseData
      • updatedRequest: { headers: SimpleHeaders; url?: string | URL }
        • headers: SimpleHeaders
        • optionalurl: string | URL

      Returns void

RegExpInput

RegExpInput: RegExp | RegExpObject

RegExpObject

RegExpObject: { regexp: RegExp } & Pick<RequestOptions, method | payload | label | userData | headers>

RequestListSourcesFunction

RequestListSourcesFunction: () => Promise<RequestListSource[]>

Type declaration

    • (): Promise<RequestListSource[]>

    • Returns Promise<RequestListSource[]>

RequestsLike

RequestsLike: AsyncIterable<Source | string> | Iterable<Source | string> | (Source | string)[]

RouterHandlerContext

RouterHandlerContext<Context, UserData>: Omit<Context, request> & { request: LoadedRequest<Request<UserData>> }

The crawling context received by a route handler, with request.userData narrowed to UserData.

Type parameters

  • Context
  • UserData: Dictionary

RouterLabel

RouterLabel<Routes>: string extends keyof Routes ? string | symbol : (keyof Routes & string) | symbol

The set of labels accepted by Router.addHandler. When the router declares a concrete route map (e.g. { PRODUCT: ...; CATEGORY: ... }), only those labels (plus symbols) are allowed — unknown labels become a compile-time error. When the map is left open (the default Record<string, ...>), any string or symbol label is accepted, preserving the original behaviour.

Type parameters

  • Routes: Record<keyof Routes, Dictionary>

RouterRoutes

RouterRoutes<Context, Routes>: { [ Label in keyof Routes ]: (ctx) => Awaitable<void> }

Type parameters

  • Context
  • Routes: Record<keyof Routes, Dictionary>

RouteSchemas

RouteSchemas: Record<string, StandardSchemaV1> & { [defaultRoute]?: StandardSchemaV1 }

A map of request labels to a Standard Schema (Zod, Valibot, ArkType, …) validating that label's request.userData. Pass it to Router.create or a createXRouter factory to derive the per-label request.userData types and validate them at runtime. The optional defaultRoute key registers a schema for requests handled by the default route.

RoutesFromSchemas

RoutesFromSchemas<Schemas>: { [ Label in Extract<keyof Schemas, string> ]: StandardSchemaV1.InferOutput<Schemas[Label]> extends Dictionary ? StandardSchemaV1.InferOutput<Schemas[Label]> : Dictionary }

Derives a route map (label → userData type) from a RouteSchemas map by inferring each schema's output type. Outputs that are not object-shaped fall back to a plain Dictionary. The defaultRoute schema drives runtime validation only, so it is excluded from the typed route map.

Type parameters

SkippedRequestCallback

SkippedRequestCallback: (args) => Awaitable<void>

Type declaration

SkippedRequestReason

SkippedRequestReason: robotsTxt | limit | enqueueLimit | filters | redirect | depth

Source

Source: (Partial<RequestOptions> & { regex?: RegExp; requestsFromUrl?: string }) | Request

UrlPatternObject

UrlPatternObject: { glob?: string; regexp?: RegExp } & Pick<RequestOptions, method | payload | label | userData | headers>

constBLOCKED_STATUS_CODES

BLOCKED_STATUS_CODES: number[] = ...

constdefaultRoute

defaultRoute: unique symbol = ...

The key of the default route — the fallback handler registered via Router.addDefaultHandler. Use it in a RouteSchemas map to register a schema that validates the userData of every request that falls through to the default handler (i.e. whose label has no route of its own).

externalconstlog

log: Log

constMAX_POOL_SIZE

MAX_POOL_SIZE: 1000 = 1000

constPERSIST_STATE_KEY

PERSIST_STATE_KEY: SDK_SESSION_POOL_STATE = 'SDK_SESSION_POOL_STATE'
Page Options