Skip to main content
Version: 4.0 (RC)

Custom logger

Crawlee uses @apify/log as its default logging library, but you can replace it with any logger you prefer, such as Winston or Pino. This is done by implementing a small adapter and passing it to the crawler.

Creating an adapter​

All Crawlee logging goes through the CrawleeLogger interface. To plug in your own logger, extend the BaseCrawleeLogger abstract class and implement two methods:

  • logWithLevel(level, message, data) — dispatches a log message to your logging library. The level parameter uses LogLevel constants (ERROR = 1, SOFT_FAIL = 2, WARNING = 3, INFO = 4, DEBUG = 5, PERF = 6). Map these to your logger's native levels. The message is a human-readable string, and data is an optional Record<string, unknown> with structured context (e.g. { url, statusCode }) — pass it to your logger as metadata or structured fields.
  • createChild(options) — returns a new child logger instance scoped to a specific component. Crawlee calls this internally to give each subsystem (e.g. CheerioCrawler, AutoscaledPool, SessionPool) its own identifiable logger. The options parameter is a CrawleeLoggerOptions object with a single field: prefix — a string label prepended to each log line from that component.

All other methods (error, warning, info, debug, exception, perf, etc.) are derived automatically from logWithLevel — you don't need to implement them.

Level filtering

logWithLevel() is called for every log message, regardless of the configured level. Level filtering is the responsibility of the underlying logging library (e.g. Winston's level option or Pino's level setting). This means your adapter doesn't need to check log levels — just forward everything and let the library decide what to output.

Injecting the logger​

There are two ways to inject a custom logger: per-crawler and globally.

Per-crawler logger​

Pass your adapter via the logger option in the crawler constructor. When a logger is provided, the crawler creates its own isolated ServiceLocator instance, so the custom logger is used by all internal components of that crawler (autoscaling, session pool, statistics, etc.):

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
    logger: new WinstonAdapter(winstonLogger),
    async requestHandler({ log }) {
        // `log` is a child of your custom logger, with prefix set to the crawler class name
        log.info('Hello from my custom logger!');
    },
});

The same logger is available as crawler.log outside of the request handler, for example when setting up routes.

Global logger via service locator​

Instead of passing the logger to each crawler individually, you can set it globally via the serviceLocator. This is useful when you run multiple crawlers and want them all to use the same logging backend:

import { serviceLocator, CheerioCrawler, PlaywrightCrawler } from 'crawlee';

// Set the logger globally — must be done before creating any crawlers
serviceLocator.setLogger(new WinstonAdapter(winstonLogger));

// Both crawlers will use the Winston logger
const cheerioCrawler = new CheerioCrawler({ /* ... */ });
const playwrightCrawler = new PlaywrightCrawler({ /* ... */ });
warning

serviceLocator.setLogger() must be called before any crawler is created. Once a logger has been retrieved from the service locator (which happens during crawler construction), it cannot be replaced — an error will be thrown.

Full examples​

import { CheerioCrawler, BaseCrawleeLogger, LogLevel } from 'crawlee';
import type { CrawleeLogger, CrawleeLoggerOptions } from 'crawlee';
import winston from 'winston';

// Map Crawlee log levels to Winston levels
const CRAWLEE_TO_WINSTON: Record<number, string> = {
    [LogLevel.ERROR]: 'error',
    [LogLevel.SOFT_FAIL]: 'warn',
    [LogLevel.WARNING]: 'warn',
    [LogLevel.INFO]: 'info',
    [LogLevel.DEBUG]: 'debug',
    [LogLevel.PERF]: 'debug',
};

class WinstonAdapter extends BaseCrawleeLogger {
    constructor(
        private logger: winston.Logger,
        options?: Partial<CrawleeLoggerOptions>,
    ) {
        super(options);
    }

    logWithLevel(level: number, message: string, data?: Record<string, unknown>): void {
        const winstonLevel = CRAWLEE_TO_WINSTON[level] ?? 'info';
        this.logger.log(winstonLevel, message, data);
    }

    protected createChild(options: Partial<CrawleeLoggerOptions>): CrawleeLogger {
        return new WinstonAdapter(this.logger.child({ prefix: options.prefix }), { ...this.getOptions(), ...options });
    }
}

// Create a Winston logger with your preferred configuration
const winstonLogger = winston.createLogger({
    level: 'debug',
    format: winston.format.combine(
        winston.format.colorize(),
        winston.format.timestamp(),
        winston.format.printf(({ level, message, timestamp, prefix }) => {
            const tag = prefix ? `[${prefix}] ` : '';
            return `${timestamp} ${level}: ${tag}${message}`;
        }),
    ),
    transports: [new winston.transports.Console()],
});

// Pass the adapter to the crawler via the `logger` option
const crawler = new CheerioCrawler({
    logger: new WinstonAdapter(winstonLogger),
    async requestHandler({ request, $, log }) {
        log.info(`Processing ${request.url}`);
        const title = $('title').text();
        log.debug('Page title extracted', { title });
    },
});

await crawler.run(['https://crawlee.dev']);