Configuration
Index
Constructors
constructor
Creates new
Configuration
instance with provided options. Env vars will have precedence over those.Parameters
options: ConfigurationOptions = {}
Returns Configuration
Properties
publicreadonlystorageManagers
Methods
get
Returns configured value. First checks the environment variables, then provided configuration, fallbacks to the
defaultValue
argument if provided, otherwise uses the default value as described in the above section.Type parameters
- T: keyof ConfigurationOptions
- U: undefined | string | number | boolean | StorageClient | Dictionary | EventManager | (radix?: number) => string | (fractionDigits?: number) => string | (fractionDigits?: number) => string | (precision?: number) => string | () => number | ({ (locales?: string | string[], options?: NumberFormatOptions): string; (locales?: LocalesArgument, options?: NumberFormatOptions): string })
Parameters
key: T
optionaldefaultValue: U
Returns U
getEventManager
Returns EventManager
set
Sets value for given option. Only affects this
Configuration
instance, the value will not be propagated down to the env var. To reset a value, we can omit thevalue
argument or passundefined
there.Parameters
key: keyof ConfigurationOptions
optionalvalue: any
Returns void
useEventManager
Parameters
events: EventManager
Returns void
useStorageClient
Parameters
client: StorageClient
Returns void
staticgetEventManager
Gets default EventManager instance.
Returns EventManager
staticgetGlobalConfig
Returns the global configuration instance. It will respect the environment variables.
Returns Configuration
staticgetStorageClient
Gets default StorageClient instance.
Returns StorageClient
staticresetGlobalState
Resets global configuration instance. The default instance holds configuration based on env vars, if we want to change them, we need to first reset the global state. Used mainly for testing purposes.
Returns void
staticset
Sets value for given option. Only affects the global
Configuration
instance, the value will not be propagated down to the env var. To reset a value, we can omit thevalue
argument or passundefined
there.Parameters
key: keyof ConfigurationOptions
optionalvalue: any
Returns void
staticuseStorageClient
Parameters
client: StorageClient
Returns void
Configuration
is a value object holding Crawlee configuration. By default, there is a global singleton instance of this class available viaConfiguration.getGlobalConfig()
. Places that depend on a configurable behaviour depend on this class, as they have the global instance as the default value.Using global configuration:
Using custom configuration:
The configuration provided via environment variables always takes precedence. We can also define the
crawlee.json
file in the project root directory which will serve as a baseline, so the options provided in constructor will override those. In other words, the precedence is:Supported Configuration Options
memoryMbytes
CRAWLEE_MEMORY_MBYTES
logLevel
CRAWLEE_LOG_LEVEL
headless
CRAWLEE_HEADLESS
true
defaultDatasetId
CRAWLEE_DEFAULT_DATASET_ID
'default'
defaultKeyValueStoreId
CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID
'default'
defaultRequestQueueId
CRAWLEE_DEFAULT_REQUEST_QUEUE_ID
'default'
persistStateIntervalMillis
CRAWLEE_PERSIST_STATE_INTERVAL_MILLIS
60_000
purgeOnStart
CRAWLEE_PURGE_ON_START
true
persistStorage
CRAWLEE_PERSIST_STORAGE
true
Advanced Configuration Options
inputKey
CRAWLEE_INPUT_KEY
'INPUT'
xvfb
CRAWLEE_XVFB
chromeExecutablePath
CRAWLEE_CHROME_EXECUTABLE_PATH
defaultBrowserPath
CRAWLEE_DEFAULT_BROWSER_PATH
disableBrowserSandbox
CRAWLEE_DISABLE_BROWSER_SANDBOX
availableMemoryRatio
CRAWLEE_AVAILABLE_MEMORY_RATIO
0.25