ContextPipeline

Encapsulates the logic of gradually enhancing the crawling context with additional information and utilities.

The enhancement is done by a chain of middlewares that are added to the pipeline after it's creation.

Index

Methods

• __call__
• __init__
• compose

Methods

__call__

• async __call__(*, crawling_context, final_context_consumer): None

• Run a crawling context through the middleware chain and pipe it into a consumer function.

  Exceptions from the consumer function are wrapped together with the final crawling context.

  ---

  Parameters

  • optionalkeyword-onlycrawling_context: BasicCrawlingContext

  • optionalkeyword-onlyfinal_context_consumer: Callable[[TCrawlingContext], Awaitable[None]]

  Returns None

__init__

• __init__(*, _middleware, _parent): None

• Parameters

  • optionalkeyword-only_middleware: Callable[[TCrawlingContext], AsyncGenerator[TMiddlewareCrawlingContext, None]] | None = None

  • optionalkeyword-only_parent: ContextPipeline[BasicCrawlingContext] | None = None

  Returns None

compose

• compose(*, middleware): ContextPipeline[TMiddlewareCrawlingContext]

• Add a middleware to the pipeline.

  The middleware should yield exactly once, and it should yield an (optionally) extended crawling context object. The part before the yield can be used for initialization and the part after it for cleanup.

  ---

  Parameters

  • optionalkeyword-onlymiddleware: Callable[[TCrawlingContext], AsyncGenerator[TMiddlewareCrawlingContext, None]]

  Returns ContextPipeline[TMiddlewareCrawlingContext]