Skip to main content
Version: Next

Configuration

Configuration is a value object holding Crawlee configuration. By default, there is a global singleton instance of this class available via Configuration.getGlobalConfiguration(). Places that depend on a configurable behaviour depend on this class, as they have the global instance as the default value.

Using global configuration:

import { BasicCrawler, Configuration } from 'crawlee';

// Get the global configuration
const config = Configuration.getGlobalConfiguration();
// Access configuration values directly as properties
console.log(config.headless);
console.log(config.persistStateIntervalMillis);

Using custom configuration:

import { BasicCrawler, Configuration } from 'crawlee';

// Create a new configuration
const config = new Configuration({ persistStateIntervalMillis: 30_000 });
// Pass the configuration to the crawler
const crawler = new BasicCrawler({ configuration: config });

Configuration is immutable — values are set via the constructor and cannot be changed afterwards. The priority order for resolving values is (highest to lowest):

constructor options > environment variables > crawlee.json > schema defaults

Supported Configuration Options

KeyEnvironment VariableDefault Value
memoryMbytesCRAWLEE_MEMORY_MBYTES-
logLevelCRAWLEE_LOG_LEVEL-
headlessCRAWLEE_HEADLESStrue
defaultDatasetIdCRAWLEE_DEFAULT_DATASET_ID'default'
defaultKeyValueStoreIdCRAWLEE_DEFAULT_KEY_VALUE_STORE_ID'default'
defaultRequestQueueIdCRAWLEE_DEFAULT_REQUEST_QUEUE_ID'default'
persistStateIntervalMillisCRAWLEE_PERSIST_STATE_INTERVAL_MILLIS60_000
internalTimeoutMillisCRAWLEE_INTERNAL_TIMEOUT-
purgeOnStartCRAWLEE_PURGE_ON_STARTtrue
persistStorageCRAWLEE_PERSIST_STORAGEtrue
storageDirCRAWLEE_STORAGE_DIR'./storage'

Advanced Configuration Options

KeyEnvironment VariableDefault Value
inputKeyCRAWLEE_INPUT_KEY'INPUT'
xvfbCRAWLEE_XVFBfalse
chromeExecutablePathCRAWLEE_CHROME_EXECUTABLE_PATH-
defaultBrowserPathCRAWLEE_DEFAULT_BROWSER_PATH-
disableBrowserSandboxCRAWLEE_DISABLE_BROWSER_SANDBOX-
availableMemoryRatioCRAWLEE_AVAILABLE_MEMORY_RATIO0.25
containerizedCRAWLEE_CONTAINERIZED-

Hierarchy

Index

Constructors

Properties

Methods

Constructors

constructor

  • Creates new Configuration instance with provided options. Constructor options take precedence over environment variables, which take precedence over crawlee.json values, which take precedence over schema defaults.

    Parameters

    • options: FieldsInput<{ availableMemoryRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; chromeExecutablePath: ConfigField<ZodOptional<ZodString>>; containerized: ConfigField<ZodOptional<ZodPreprocess<ZodBoolean>>>; defaultBrowserPath: ConfigField<ZodOptional<ZodString>>; defaultDatasetId: ConfigField<ZodDefault<ZodString>>; defaultKeyValueStoreId: ConfigField<ZodDefault<ZodString>>; defaultRequestQueueId: ConfigField<ZodDefault<ZodString>>; disableBrowserSandbox: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; headless: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; inputKey: ConfigField<ZodDefault<ZodString>>; internalTimeoutMillis: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>; logLevel: ConfigField<ZodOptional<ZodPreprocess<ZodEnum<typeof LogLevel>>>>; maxUsedCpuRatio: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; memoryMbytes: ConfigField<ZodOptional<ZodPreprocess<ZodNumber>>>; persistStateIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; persistStorage: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; purgeOnStart: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>>; storageDir: ConfigField<ZodDefault<ZodString>>; systemInfoIntervalMillis: ConfigField<ZodDefault<ZodPreprocess<ZodNumber>>>; xvfb: ConfigField<ZodDefault<ZodPreprocess<ZodBoolean>>> }> = {}

    Returns Configuration

Properties

inheritedavailableMemoryRatio

availableMemoryRatio: number = ...

inheritedchromeExecutablePath

chromeExecutablePath: undefined | string = ...

inheritedcontainerized

containerized: undefined | boolean = ...

inheriteddefaultBrowserPath

defaultBrowserPath: undefined | string = ...

inheriteddefaultDatasetId

defaultDatasetId: string = ...

inheriteddefaultKeyValueStoreId

defaultKeyValueStoreId: string = ...

inheriteddefaultRequestQueueId

defaultRequestQueueId: string = ...

inheriteddisableBrowserSandbox

disableBrowserSandbox: boolean = ...

inheritedheadless

headless: boolean = ...

inheritedinputKey

inputKey: string = ...

inheritedinternalTimeoutMillis

internalTimeoutMillis: undefined | number = ...

Internal safety-net timeout for a single request, in milliseconds. When unset the crawler derives it from the request handler timeout (twice it, and never below 5 minutes).

inheritedlogLevel

logLevel: undefined | OFF | ERROR | SOFT_FAIL | WARNING | INFO | DEBUG | PERF = ...

inheritedmaxUsedCpuRatio

maxUsedCpuRatio: number = ...

inheritedmemoryMbytes

memoryMbytes: undefined | number = ...

inheritedpersistStateIntervalMillis

persistStateIntervalMillis: number = ...

inheritedpersistStorage

persistStorage: boolean = ...

inheritedpurgeOnStart

purgeOnStart: boolean = ...

inheritedstorageDir

storageDir: string = ...

inheritedsystemInfoIntervalMillis

systemInfoIntervalMillis: number = ...

inheritedxvfb

xvfb: boolean = ...

Methods

staticgetGlobalConfiguration

  • Returns the global configuration instance. It will respect the environment variables.

    Delegates to the global ServiceLocator, making it the single source of truth for service management.

    Returns Configuration

Page Options