ConfigurationOptions

Index

Properties

• availableMemoryRatio
• chromeExecutablePath
• defaultBrowserPath
• defaultDatasetId
• defaultKeyValueStoreId
• defaultRequestQueueId
• disableBrowserSandbox
• eventManager
• headless
• inputKey
• logLevel
• maxUsedCpuRatio
• memoryMbytes
• persistStateIntervalMillis
• persistStorage
• purgeOnStart
• storageClient
• storageClientOptions
• systemInfoIntervalMillis
• xvfb

Properties

optionalavailableMemoryRatio

availableMemoryRatio?: number = 0.25

Sets the ratio, defining the amount of system memory that could be used by the AutoscaledPool. When the memory usage is more than the provided ratio, the memory is considered overloaded.

Alternative to CRAWLEE_AVAILABLE_MEMORY_RATIO environment variable.

optionalchromeExecutablePath

chromeExecutablePath?: string

Defines a path to Chrome executable.

Alternative to CRAWLEE_CHROME_EXECUTABLE_PATH environment variable.

optionaldefaultBrowserPath

defaultBrowserPath?: string

Defines a path to default browser executable.

Alternative to CRAWLEE_DEFAULT_BROWSER_PATH environment variable.

optionaldefaultDatasetId

defaultDatasetId?: string = ‘default’

Default dataset id.

Alternative to CRAWLEE_DEFAULT_DATASET_ID environment variable.

optionaldefaultKeyValueStoreId

defaultKeyValueStoreId?: string = ‘default’

Default key-value store id.

Alternative to CRAWLEE_DEFAULT_KEY_VALUE_STORE_ID environment variable.

optionaldefaultRequestQueueId

defaultRequestQueueId?: string = ‘default’

Default request queue id.

Alternative to CRAWLEE_DEFAULT_REQUEST_QUEUE_ID environment variable.

optionaldisableBrowserSandbox

disableBrowserSandbox?: boolean

Defines whether to disable browser sandbox by adding --no-sandbox flag to launchOptions.

Alternative to CRAWLEE_DISABLE_BROWSER_SANDBOX environment variable.

optionaleventManager

eventManager?: EventManager = EventManager

Defines the Event Manager to be used.

optionalheadless

headless?: boolean = true

Defines whether web browsers launched by Crawlee will run in the headless mode.

Alternative to CRAWLEE_HEADLESS environment variable.

optionalinputKey

inputKey?: string = ‘INPUT’

Defines the default input key, i.e. the key that is used to get the crawler input value from the default KeyValueStore associated with the current crawler run.

Alternative to CRAWLEE_INPUT_KEY environment variable.

optionallogLevel

logLevel?: LogLevel | (radix?: number) => string | (fractionDigits?: number) => string | (fractionDigits?: number) => string | (precision?: number) => string | () => number | ({ (locales?: string | string[], options?: NumberFormatOptions): string; (locales?: LocalesArgument, options?: NumberFormatOptions): string }) = LogLevel | (radix?: number) => string | (fractionDigits?: number) => string | (fractionDigits?: number) => string | (precision?: number) => string | () => number | ({ (locales?: string | string[], options?: NumberFormatOptions): string; (locales?: LocalesArgument, options?: NumberFormatOptions): string })

Sets the log level to the given value.

Alternative to CRAWLEE_LOG_LEVEL environment variable.

optionalmaxUsedCpuRatio

maxUsedCpuRatio?: number = 0.95

Sets the ratio, defining the maximum CPU usage. When the CPU usage is higher than the provided ratio, the CPU is considered overloaded.

optionalmemoryMbytes

memoryMbytes?: number

Sets the amount of system memory in megabytes to be used by the AutoscaledPool. By default, the maximum memory is set to one quarter of total system memory.

Alternative to CRAWLEE_MEMORY_MBYTES environment variable.

optionalpersistStateIntervalMillis

persistStateIntervalMillis?: number = 60_000

Defines the interval of emitting the persistState event.

Alternative to CRAWLEE_PERSIST_STATE_INTERVAL_MILLIS environment variable.

optionalpersistStorage

persistStorage?: boolean

Defines whether the storage client used should persist the data it stores.

Alternative to CRAWLEE_PERSIST_STORAGE environment variable.

optionalpurgeOnStart

purgeOnStart?: boolean = true

Defines whether to purge the default storage folders before starting the crawler run.

Alternative to CRAWLEE_PURGE_ON_START environment variable.

optionalstorageClient

storageClient?: StorageClient = StorageClient

Defines storage client to be used.

optionalstorageClientOptions

storageClientOptions?: Dictionary

Could be used to adjust the storage client behavior e.g. MemoryStorageOptions could be used to adjust the MemoryStorage behavior.

optionalsystemInfoIntervalMillis

systemInfoIntervalMillis?: number = 60_000

Defines the interval of emitting the systemInfo event.

optionalxvfb

xvfb?: boolean = false

Defines whether to run X virtual framebuffer on the web browsers launched by Crawlee.

Alternative to CRAWLEE_XVFB environment variable.