Skip to main content
Version: Next

Router <Context, Routes>

Simple router that works based on request labels. This instance can then serve as a requestHandler of your crawler.

import { Router, CheerioCrawler, CheerioCrawlingContext } from 'crawlee';

const router = Router.create<CheerioCrawlingContext>();

// we can also use factory methods for specific crawling contexts, the above equals to:
// import { createCheerioRouter } from 'crawlee';
// const router = createCheerioRouter();

router.addHandler('label-a', async (ctx) => {
   ctx.log.info('...');
});
router.addDefaultHandler(async (ctx) => {
   ctx.log.info('...');
});

const crawler = new CheerioCrawler({
    requestHandler: router,
});
await crawler.run();

Alternatively we can use the default router instance from crawler object:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler();

crawler.router.addHandler('label-a', async (ctx) => {
   ctx.log.info('...');
});
crawler.router.addDefaultHandler(async (ctx) => {
   ctx.log.info('...');
});

await crawler.run();

For convenience, we can also define the routes right when creating the router:

import { CheerioCrawler, createCheerioRouter } from 'crawlee';
const crawler = new CheerioCrawler({
    requestHandler: createCheerioRouter({
        'label-a': async (ctx) => { ... },
        'label-b': async (ctx) => { ... },
    })},
});
await crawler.run();

Middlewares are also supported via the router.use method. There can be multiple middlewares for a single router, they will be executed sequentially in the same order as they were registered.

crawler.router.use(async (ctx) => {
   ctx.log.info('...');
});

To get request.userData typed per label, declare a route map and pass it as the second type argument. The label passed to Router.addHandler then drives the type of request.userData, and unknown labels are rejected at compile time:

import { createCheerioRouter, CheerioCrawlingContext } from 'crawlee';

interface Routes {
    PRODUCT: { sku: string; price: number };
    CATEGORY: { categoryId: string };
}

const router = createCheerioRouter<CheerioCrawlingContext, Routes>();

router.addHandler('PRODUCT', async ({ request }) => {
    request.userData.sku;   // string
    request.userData.price; // number
});

router.addHandler('TYPO', async () => {}); // compile error: not a known label

Passing a Standard Schema per label instead of a plain type both infers the request.userData types and validates them at runtime — when the request is handled, and when it is added to the crawler (crawler.addRequests, context.addRequests, enqueueLinks). A failing request throws a RequestValidationError.

import { z } from 'zod';
import { createCheerioRouter } from 'crawlee';

const router = createCheerioRouter({
    PRODUCT: z.object({ sku: z.string(), price: z.number() }),
    CATEGORY: z.object({ categoryId: z.string() }),
});

router.addHandler('PRODUCT', async ({ request }) => {
    request.userData.price; // number, inferred from the schema and validated at runtime
});

A single route can take longer than the rest without raising the crawler-wide requestHandlerTimeoutSecs for everything - pass a per-route timeout as the last argument:

// LIST pages scroll through a lot of content, DETAIL pages are quick
router.addHandler('LIST', async (ctx) => { ... }, { requestHandlerTimeoutSecs: 120 });
router.addHandler('DETAIL', async (ctx) => { ... }); // keeps the crawler's default

When the time a route needs is only apparent once it is already running, call context.extendTimeout from inside the handler:

router.addHandler('LIST', async ({ page, extendTimeout }) => {
    const pageCount = await countPages(page);
    extendTimeout(pageCount * 10); // ask for 10 more seconds per page
    await scrapeAllPages(page);
});

Hierarchy

Index

Methods

Methods

addDefaultHandler

  • addDefaultHandler<UserData>(handler, options): void

  • Registers default route handler. As a fallback it can receive any request (including labels not declared in the route map). When the router was created with a defaultRoute schema, request.userData is typed from it; otherwise it defaults to the context's (loosely typed) userData. Pass an explicit UserData type argument to narrow it. Pass options to give the default route its own requestHandlerTimeoutSecs, overriding the crawler's default for requests that fall through to it.

    Parameters

    Returns void

addHandler

  • addHandler<Label>(label, handler, options): void
  • addHandler<UserData>(label, handler, options): void

  • Registers new route handler for given label. When the router declares a route map, the label is restricted to the declared labels and request.userData is typed accordingly. Pass options to give this route its own requestHandlerTimeoutSecs, overriding the crawler's default for requests with this label.

    Parameters

    • label: Label
    • handler: (ctx) => Awaitable<void>
      • optionaloptions: RouteOptions

      Returns void

    getHandler

    • getHandler(label): (ctx) => Awaitable<void>

    • Returns route handler for given label. If no label is provided, the default request handler will be returned.

      Parameters

      • optionallabel: string | symbol

      Returns (ctx) => Awaitable<void>

        • (ctx): Awaitable<void>

        • Parameters

          • ctx: Context

          Returns Awaitable<void>

    getMaxTimeoutSecs

    • getMaxTimeoutSecs(): undefined | number

    • The longest requestHandlerTimeoutSecs any route asked for, or undefined when no route overrides it. The crawler needs an upper bound up front, before it knows which routes a run will actually hit.

      Returns undefined | number

    getTimeoutSecs

    • getTimeoutSecs(label): undefined | number

    • Returns the requestHandlerTimeoutSecs registered for a label, or undefined when the route did not override it and the crawler's own timeout should apply. Falls back to the default route the same way getHandler does, so a label with no route of its own inherits whatever the default route asked for. Used by the crawler; not meant to be called directly.

      Parameters

      • optionallabel: string | symbol

      Returns undefined | number

    use

    • use(middleware): void

    • Registers a middleware that will be fired before the matching route handler. Multiple middlewares can be registered, they will be fired in the same order.

      Parameters

      • middleware: (ctx) => Awaitable<void>

        Returns void

      staticcreate

      • Creates new router instance. This instance can then serve as a requestHandler of your crawler.

        import { Router, CheerioCrawler, CheerioCrawlingContext } from 'crawlee';

        const router = Router.create<CheerioCrawlingContext>();
        router.addHandler('label-a', async (ctx) => {
           ctx.log.info('...');
        });
        router.addDefaultHandler(async (ctx) => {
           ctx.log.info('...');
        });

        const crawler = new CheerioCrawler({
            requestHandler: router,
        });
        await crawler.run();

        Parameters

        Returns RouterHandler<Context, Routes>

      Page Options