Skip to main content
Version: Next

@crawlee/core

Core set of classes required for Crawlee.

The crawlee package consists of several smaller packages, released separately under @crawlee namespace:

Installing Crawlee​

Most of the Crawlee packages are extending and reexporting each other, so it's enough to install just the one you plan on using, e.g. @crawlee/playwright if you plan on using playwright - it already contains everything from the @crawlee/browser package, which includes everything from @crawlee/basic, which includes everything from @crawlee/core.

If we don't care much about additional code being pulled in, we can just use the crawlee meta-package, which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

npm install crawlee

Or if all we need is cheerio support, we can install only @crawlee/cheerio.

npm install @crawlee/cheerio

When using playwright or puppeteer, we still need to install those dependencies explicitly - this allows the users to be in control of which version will be used.

npm install crawlee playwright
# or npm install @crawlee/playwright playwright

Alternatively we can also use the crawlee meta-package which contains (re-exports) most of the @crawlee/* packages, and therefore contains all the crawler classes.

Sometimes you might want to use some utility methods from @crawlee/utils, so you might want to install that as well. This package contains some utilities that were previously available under Apify.utils. Browser related utilities can be also found in the crawler packages (e.g. @crawlee/playwright).

Index

Result Stores

Scaling

Sources

Other

Other

ConfigurationInput

ConfigurationInput: FieldsInput<typeof crawleeConfigFields>

ConfigurationOptions

ConfigurationOptions: ConfigurationInput

Deprecated - Use ConfigurationInput instead.

EnqueueStrategyOption

EnqueueStrategyOption: EnqueueStrategy | all | same-domain | same-hostname | same-origin

The strategy option accepted by ExtractLinksOptions and EnqueueUrlsOptions.

EventTypeName

EventTypeName: EventType | systemInfo | persistState | migrating | aborting | exit | statusMessage

ExplicitStorageIdentifier

ExplicitStorageIdentifier: { alias?: never; id: string; name?: never } | { alias?: never; id?: never; name: string } | { alias: string; id?: never; name?: never }

A storage identifier where exactly one of id, name, or alias is specified. Produced by resolveStorageIdentifier from ambiguous user input.

FieldsInput

FieldsInput<F>: { [ K in keyof F ]?: z.output<F[K][schema]> }

Type parameters

FieldsOutput

FieldsOutput<F>: { [ K in keyof F ]: z.output<F[K][schema]> }

Type parameters

JournalEntry

JournalEntry: DatasetJournalEntry | KeyValueStoreJournalEntry | RequestQueueJournalEntry

PacingScope

PacingScope: LiteralUnion<hostname | registrableDomain, string>

How much of the URL space a PacingSignal covers.

Open on purpose: 'hostname' and 'registrableDomain' are what Crawlee's own reporters send and what ThrottlingRequestManager understands, but any string is accepted, so a pacer keyed on something else can be reported to in its own vocabulary.

PacingSignal

PacingSignal: { reason: rateLimited; scope?: PacingScope; url: string; waitMs?: number } | { intervalMs: number; reason: minInterval; scope: PacingScope; url: string } | { intervalMs: number; reason: minIntervalEverywhere; scope: PacingScope }

Something said about the pace requests should go out at, reported to a request manager through IRequestManager.recordPacingSignal.

One shape rather than a method per channel: a pacing manager switches on reason, a wrapping one forwards the value without knowing what is in it, and a new kind of signal costs the interface nothing. The url travels inside the value because the crawl-wide variant has none. Nothing here names the mechanism a signal came from - status codes, headers and robots.txt are the crawler's business - and every delay is in milliseconds.

Scope

A manager may apply a signal to a wider scope than it was given - a floor that holds for one host still holds when a whole site is paced by it - but never to a narrower one, which would leave some of the URLs the signal covers running unpaced. A manager that can only do the latter, or that does not recognise the scope at all, MUST throw rather than quietly under-apply it.

RequestListSourcesFunction

RequestListSourcesFunction: () => Promise<RequestListSource[]>

Type declaration

    • (): Promise<RequestListSource[]>

    • Returns Promise<RequestListSource[]>

RequestLoaderStatus

RequestLoaderStatus: Exclude<RequestSourceStatus, { status: stalled }>

Loaders never stall — only a manager that paces its own dispatch can.

RequestsLike

RequestsLike: AsyncIterable<Source | string> | Iterable<Source | string> | (Source | string)[]

RequestSourceStatus

RequestSourceStatus: { status: ready } | { readyAt?: number; status: waiting } | { reason: string; status: stalled } | { status: finished }

A request source's own availability, in a single answer.

  • ready — the next IRequestLoader.fetchNextRequest is expected to hand something over.
  • waiting — nothing to fetch right now, but the source is not done: requests are in progress, are being added in the background, or are held back until readyAt.
  • stalled — the source holds requests it cannot make progress on. Only a manager that paces its own dispatch can reach this; see ThrottlingRequestManager.
  • finished — everything has been handled.

ResolvedConfigValues

ResolvedConfigValues: FieldsOutput<typeof crawleeConfigFields>

SkippedRequestReason

SkippedRequestReason: robotsTxt | limit | enqueueLimit | filters | transform | redirect | depth

Source

Source: (Partial<RequestOptions> & { regex?: RegExp; requestsFromUrl?: string }) | CrawleeRequest

StateConversion

StateConversion<TFrom, TTo>: (value) => Awaitable<TTo> | StandardSchemaV1<TFrom, TTo>

One direction of the conversion between the state model and its persisted form - either a plain function, or a Standard Schema whose validated output is the result.

A schema that fails to validate makes RecoverableState throw a StateValidationError. Zod codecs work directly, as their validation is the decode direction; use (state) => codec.encode(state) for the other one.

Type parameters

  • TFrom
  • TTo

StorageIdentifier

StorageIdentifier: { alias?: never; id: string; name?: never } | { alias?: never; id?: never; name: string } | { alias: string; id?: never; name?: never } | { alias?: never; id?: never; name?: never }

Identifies a storage by its ID, name, or alias. At most one may be provided.

  • { id } — open a pre-existing storage by its unique ID.
  • { name } — open or create a globally named storage (persists across runs). The name default is reserved: it resolves to the default storage, and is emptied on start along with it.
  • { alias } — open or create a run-scoped unnamed storage identified by this alias. The alias is used locally (e.g. as a directory name or cache key) but the storage itself has no persistent name. Use this for non-default unnamed storages. Like the default storage, an aliased one is emptied on start unless purgeOnStart is disabled.
  • {} / omitted — open the default storage.

StorageTransactionState

StorageTransactionState: open | committing | committed | failed | rolledBack

StorageWriteMode

StorageWriteMode: deferred | writeThrough

Governs whether writes of a given storage type performed inside a StorageTransaction are applied immediately (writeThrough) or recorded and replayed on commit (deferred).

SyncStateConversion

SyncStateConversion<TFrom, TTo>: (value) => TTo | StandardSchemaV1<TFrom, TTo>

A StateConversion for a caller that cannot await one - Statistics, whose toJSON() is synchronous, being the reason this exists.

Only the function arm can be narrowed here: a Standard Schema is free to validate asynchronously, so a schema that does is rejected when it runs rather than when it is passed.

Type parameters

  • TFrom
  • TTo

constcoerceBoolean

coerceBoolean: ZodPreprocess<ZodBoolean, unknown> = ...

Zod preprocessor treating '0' and 'false' as falsy.

constcoerceNumber

coerceNumber: ZodPreprocess<ZodNumber, unknown> = ...

constcrawleeConfigFields

crawleeConfigFields: { availableMemoryRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>; chromeExecutablePath: ConfigField<ZodOptional<ZodString>>; containerized: ConfigField<ZodOptional<ZodPreprocess<ZodBoolean, unknown>>>; defaultBrowserPath: ConfigField<ZodOptional<ZodString>>; defaultDatasetId: ConfigField<ZodDefault<ZodString>>; defaultKeyValueStoreId: ConfigField<ZodDefault<ZodString>>; defaultRequestQueueId: ConfigField<ZodDefault<ZodString>>; disableBrowserSandbox: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>; headless: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>; inputKey: ConfigField<ZodDefault<ZodString>>; internalTimeoutMillis: ConfigField<ZodOptional<ZodPreprocess<ZodNumber, unknown>>>; logLevel: ConfigField<ZodOptional<ZodPreprocess<ZodEnum<typeof LogLevel>, unknown>>>; maxUsedCpuRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>; memoryMbytes: ConfigField<ZodOptional<ZodPreprocess<ZodNumber, unknown>>>; persistStateIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>; persistStorage: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>; purgeOnStart: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>; storageDir: ConfigField<ZodDefault<ZodString>>; systemInfoIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>; xvfb: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>> } = ...

Type declaration

  • availableMemoryRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>
  • chromeExecutablePath: ConfigField<ZodOptional<ZodString>>
  • containerized: ConfigField<ZodOptional<ZodPreprocess<ZodBoolean, unknown>>>
  • defaultBrowserPath: ConfigField<ZodOptional<ZodString>>
  • defaultDatasetId: ConfigField<ZodDefault<ZodString>>
  • defaultKeyValueStoreId: ConfigField<ZodDefault<ZodString>>
  • defaultRequestQueueId: ConfigField<ZodDefault<ZodString>>
  • disableBrowserSandbox: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>
  • headless: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>
  • inputKey: ConfigField<ZodDefault<ZodString>>
  • internalTimeoutMillis: ConfigField<ZodOptional<ZodPreprocess<ZodNumber, unknown>>>

    Internal safety-net timeout for a single request, in milliseconds. When unset the crawler derives it from the request handler timeout (twice it, and never below 5 minutes).

  • logLevel: ConfigField<ZodOptional<ZodPreprocess<ZodEnum<typeof LogLevel>, unknown>>>
  • maxUsedCpuRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>
  • memoryMbytes: ConfigField<ZodOptional<ZodPreprocess<ZodNumber, unknown>>>
  • persistStateIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>
  • persistStorage: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>
  • purgeOnStart: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>
  • storageDir: ConfigField<ZodDefault<ZodString>>
  • systemInfoIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber, unknown>>>
  • xvfb: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean, unknown>>>

externalconstlog

log: Log

constserviceLocator

serviceLocator: ServiceLocatorInterface = ...
Page Options