Skip to main content
Version: Next

Migrating from Scrapy

Scrapy and Crawlee solve the same problem, so most of what you know carries over. The concepts line up almost one to one, and the selector code often moves across without changes. Scrapy parses HTML with Parsel, and so does Crawlee's ParselCrawler. Your response.css() and response.xpath() queries keep working on context.selector.

This guide maps Scrapy concepts to their Crawlee equivalents and rewrites a small spider step by step. The examples use ParselCrawler because it's the closest match for a Scrapy spider. The other HTTP option is BeautifulSoupCrawler, and the choice between them is covered in the HTTP crawlers guide. For pages that need a browser, Crawlee has PlaywrightCrawler, covered in the Playwright crawler guide.

The Crawlee examples cap each run with max_requests_per_crawl to keep the demos short. The Scrapy equivalent is the CLOSESPIDER_PAGECOUNT setting from the built-in CloseSpider extension.

Installation​

Install Crawlee with the Parsel extra:

pip install 'crawlee[parsel]'

For the browser examples, also install Playwright:

pip install 'crawlee[playwright]'
playwright install

Conceptual differences​

Both frameworks give you a request scheduler, filtering of duplicate requests, retries, and a way to pull data out of pages. The way you wire those pieces together differs.

  • Scrapy starts spiders through its own scrapy crawl command, so tracing the program flow and debugging it step by step takes extra setup. A Crawlee project is a regular Python script. You run it with python main.py and step through it in any debugger.
  • Browser rendering, proxy rotation, session management, and storage are part of the framework. In Scrapy you reach for scrapy-playwright, proxy middlewares, and feed exporters. In Crawlee these are built in and share one API.
  • Scrapy runs on the Twisted reactor, and your callbacks are synchronous generators that yield items and requests. Crawlee runs on asyncio, and handlers are coroutines defined with async def that await helpers like push_data and enqueue_links. You write straight-line async/await code without a reactor.
  • A Scrapy spider is a class with a parse method and callbacks. Crawlee routes requests through a Router that dispatches them to handler functions by their label.

Mapping key concepts​

ScrapyCrawlee
scrapy.SpiderParselCrawler
start_urls / start_requests()crawler.run([...])
parse(self, response)@crawler.router.default_handler
callback=self.parse_detail@crawler.router.handler('detail')
cb_kwargs / Request.metaRequest.user_data
response.css() / response.xpath()context.selector
yield {...}push_data
scrapy.ItemA plain dict
response.follow() / yield Request(...)enqueue_links / add_requests
dont_filter=TrueRequest.from_url(always_enqueue=True)
allowed_domainsenqueue_links(strategy=...)
scrapy.FormRequestRequest.from_url(method='POST', payload=...)
Item pipelinesDataset
Downloader / spider middlewaresrouter.use(), navigation hooks, HTTP clients
settings.pyConfiguration + crawler arguments
CONCURRENT_REQUESTS / DOWNLOAD_DELAYConcurrencySettings
AutoThrottleThrottlingRequestManager
DEPTH_LIMITmax_crawl_depth
CLOSESPIDER_PAGECOUNTmax_requests_per_crawl
DOWNLOAD_TIMEOUTnavigation_timeout
ROBOTSTXT_OBEYrespect_robots_txt_file
RetryMiddlewaremax_request_retries
errbackfailed_request_handler
FEEDS / feed exportsexport_data
CrawlSpider / Rulesenqueue_links(selector=..., label=...)
HttpProxyMiddlewareProxyConfiguration
scrapy-playwrightPlaywrightCrawler
scrapy crawl myspiderpython main.py

Migrating a spider​

Here's the canonical quotes spider from the Scrapy tutorial. It reads quotes from a listing page and follows the pagination link. The Crawlee version keeps the selectors untouched. The parse method becomes the default handler, yield {...} becomes push_data, and the manual pagination becomes a single enqueue_links call.

Run on
import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        await context.push_data(
            [
                {
                    'text': quote.css('span.text::text').get(),
                    'author': quote.css('small.author::text').get(),
                    'tags': quote.css('div.tags a.tag::text').getall(),
                }
                for quote in context.selector.css('div.quote')
            ]
        )
        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Note that:

  • context.selector is the Parsel Selector that Scrapy calls response. Every CSS and XPath query works the same way.
  • enqueue_links extracts the href from matching elements, resolves it against the current URL, and schedules it. It also skips URLs already seen, so you don't reimplement the duplicate filter.
  • push_data writes to the default Dataset. There's no separate item pipeline to declare.

Routing with labels​

Scrapy routes pages by passing a callback to each request. A listing page hands its links to parse_author. In Crawlee you attach a label to the enqueued requests, then register a handler for that label. Each request goes to the matching handler, and unlabeled requests fall through to the default one. For details, see the Request router guide.

Run on
import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=50)

    # The default handler processes listing pages: the entry point and each
    # paginated page. It routes author links to a separate handler by label.
    @crawler.router.default_handler
    async def listing_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Listing {context.request.url}')

        # Enqueue author detail pages with a label. It replaces a Scrapy
        # `Request(url, callback=self.parse_author)`.
        await context.enqueue_links(selector='div.quote span a', label='author')

        # Follow the pagination link.
        await context.enqueue_links(selector='li.next a')

    # This handler runs only for requests labeled 'author'.
    @crawler.router.handler('author')
    async def author_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Author {context.request.url}')

        await context.push_data(
            {
                'name': context.selector.css('h3.author-title::text').get(),
                'born': context.selector.css('span.author-born-date::text').get(),
                'bio': context.selector.css('div.author-description::text').get(),
            }
        )

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Passing data between handlers​

A listing page often holds data you want to keep, then merge with fields from the detail page. Scrapy carries it through cb_kwargs (or Request.meta). Crawlee carries it through Request.user_data, which travels with the request and is available on context.request.user_data in the next handler.

How you attach it depends on the data. When each link carries its own value, like a price that belongs to one book, create a request per URL with Request.from_url(..., user_data=...) and add them with add_requests. That's what the Crawlee example does. When a single value applies to the whole batch, skip the loop and pass user_data straight to enqueue_links.

Run on
import asyncio
from urllib.parse import urljoin

from crawlee import Request
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def listing_handler(context: ParselCrawlingContext) -> None:
        requests: list[Request] = []

        for book in context.selector.css('article.product_pod'):
            href = book.css('h3 a::attr(href)').get()
            if href is None:
                continue

            # `user_data` travels with the request to the detail handler,
            # the same role Scrapy's `cb_kwargs` plays.
            requests.append(
                Request.from_url(
                    urljoin(context.request.url, href),
                    label='book',
                    user_data={
                        'listing_price': book.css('p.price_color::text').get(),
                    },
                )
            )

        await context.add_requests(requests)

    @crawler.router.handler('book')
    async def book_handler(context: ParselCrawlingContext) -> None:
        await context.push_data(
            {
                'title': context.selector.css('h1::text').get(),
                'listing_price': context.request.user_data.get('listing_price'),
            }
        )

    await crawler.run(['https://books.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Rule-based crawling​

Scrapy's CrawlSpider declares Rules with a LinkExtractor to follow links automatically and dispatch them to callbacks. Crawlee expresses the same intent with enqueue_links. The restrict_css and restrict_xpaths arguments map to selector, and the allow and deny patterns map to include and exclude, which accept Glob patterns or regular expressions.

Run on
import asyncio

from crawlee import Glob
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=50)

    # The default handler plays the role of CrawlSpider's `rules`. It follows the
    # pagination and enqueues each book detail page, routed by label.
    @crawler.router.default_handler
    async def listing_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Listing {context.request.url}')

        # `selector` is the `restrict_css` analog. `include` is the `allow` analog:
        # it keeps only URLs matching the given globs.
        await context.enqueue_links(
            selector='article.product_pod h3 a',
            include=[Glob('https://books.toscrape.com/catalogue/**')],
            label='book',
        )

        # Follow pagination without a label, like a `Rule` with no callback.
        await context.enqueue_links(selector='li.next a')

    # Routed by the 'book' label, like a `Rule` with `callback='parse_book'`.
    @crawler.router.handler('book')
    async def book_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Book {context.request.url}')
        await context.push_data(
            {
                'title': context.selector.css('h1::text').get(),
                'price': context.selector.css('p.price_color::text').get(),
            }
        )

    await crawler.run(['https://books.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Exporting data​

Scrapy writes results through the FEEDS setting or the -O command-line flag. Crawlee collects items in the default Dataset as you call push_data, then export_data writes the whole dataset once the run finishes. The format is chosen by the file extension, either .json or .csv.

Run on
import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')

        items = [
            {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
            }
            for quote in context.selector.css('div.quote')
        ]
        await context.push_data(items)

        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/'])

    # Export the whole dataset to a file. The format follows the extension,
    # which must be .json or .csv. It replaces Scrapy's `FEEDS` setting and
    # the `-O output.json` CLI flag.
    await crawler.export_data('quotes.json')
    await crawler.export_data('quotes.csv')


if __name__ == '__main__':
    asyncio.run(main())

The default (unnamed) Dataset stays on disk after the run finishes, but the next run clears it before starting, since Configuration.purge_on_start defaults to True. To keep data across runs, use a named dataset or turn that option off. For details, see the Storages guide. To continue an interrupted crawl from where it stopped, see the Resuming a paused crawl example.

Concurrency and throttling​

Both Scrapy and Crawlee have settings to influence the scraping throughput, but they differ in their approach. Crawlee reads ConcurrencySettings and scales concurrency automatically based on system resources, as described in the Scaling crawlers guide. Use max_tasks_per_minute to cap the request rate. It's close to DOWNLOAD_DELAY, but it limits the whole pool rather than adding a fixed delay per domain.

Run on
import asyncio

from crawlee import ConcurrencySettings
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    # `ConcurrencySettings` replaces Scrapy's `CONCURRENT_REQUESTS` and
    # `DOWNLOAD_DELAY`.
    concurrency_settings = ConcurrencySettings(
        # Start with this many parallel tasks.
        desired_concurrency=5,
        # Never run more than this many in parallel.
        max_concurrency=20,
        # Cap total throughput across the whole pool.
        max_tasks_per_minute=120,
    )

    crawler = ParselCrawler(
        concurrency_settings=concurrency_settings,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')
        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

The closest built-in analog to the AutoThrottle extension is the ThrottlingRequestManager, covered in the Request throttling guide. It differs in what drives the backoff. Scrapy's AutoThrottle adapts to measured latency, while Crawlee's manager reacts to explicit signals: HTTP 429 replies and robots.txt crawl-delay directives. It wraps a RequestQueue and applies the backoff per domain. You list the domains to watch, and the crawler reports the signals to the manager on its own.

Crawl-delay works only when the crawler fetches robots.txt. Projects generated by scrapy startproject obey robots.txt out of the box, but the default for Crawlee is to ignore it. Passing respect_robots_txt_file=True tells Crawlee to enforce the Disallow rules. Together with ThrottlingRequestManager it also reads crawl-delay, otherwise it logs a warning.

Run on
import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.request_loaders import ThrottlingRequestManager
from crawlee.storages import RequestQueue


async def main() -> None:
    # A regular request queue holds requests for non-throttled domains.
    request_queue = await RequestQueue.open()

    # `ThrottlingRequestManager` wraps the queue and adds per-domain backoff.
    # It reacts to HTTP 429 responses and `robots.txt` crawl-delay directives, which
    # makes it the closest built-in analog to Scrapy's `AutoThrottle`. The crawler
    # feeds it those signals, so you only list the domains to watch.
    request_manager = ThrottlingRequestManager(
        inner=request_queue,
        domains=['quotes.toscrape.com'],
        request_manager_opener=RequestQueue.open,
    )

    crawler = ParselCrawler(
        request_manager=request_manager,
        # Crawl-delay is only read when `robots.txt` handling is enabled.
        respect_robots_txt_file=True,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')
        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Proxies​

Scrapy rotates proxies through a middleware or a third-party package. Crawlee takes a ProxyConfiguration and rotates the URLs for you. For tiered proxies, session-bound proxies, and integration with the Apify Proxy, see the Proxy management guide.

Run on
import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration


async def main() -> None:
    # `ProxyConfiguration` replaces Scrapy's `HttpProxyMiddleware` and the
    # `scrapy-rotating-proxies` package. The URLs rotate in a round-robin fashion.
    proxy_configuration = ProxyConfiguration(
        proxy_urls=[
            'http://proxy-1.com/',
            'http://proxy-2.com/',
        ]
    )

    crawler = ParselCrawler(
        proxy_configuration=proxy_configuration,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')
        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Error handling​

Scrapy retries failed requests with RetryMiddleware and reports terminal failures through errback. Crawlee retries with max_request_retries, runs an error_handler between attempts, and calls a failed_request_handler once the retries run out. For details, see the Error handling guide.

Run on
import asyncio

from crawlee.crawlers import BasicCrawlingContext, ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(
        # Retry each failed request up to this many times (the `RetryMiddleware` analog).
        max_request_retries=3,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def handler(context: ParselCrawlingContext) -> None:
        for quote in context.selector.css('div.quote'):
            await context.push_data({'text': quote.css('span.text::text').get()})

    # Runs between retries. It can inspect or adjust the request before the next try.
    @crawler.error_handler
    async def on_error(context: BasicCrawlingContext, error: Exception) -> None:
        context.log.warning(f'Retrying {context.request.url}: {error}')

    # Runs once a request has exhausted all retries, like Scrapy's `errback`.
    @crawler.failed_request_handler
    async def on_failed(context: BasicCrawlingContext, error: Exception) -> None:
        context.log.error(f'Giving up on {context.request.url}: {error}')

    await crawler.run(['https://quotes.toscrape.com/'])


if __name__ == '__main__':
    asyncio.run(main())

Forms and login​

Scrapy submits forms with FormRequest, which encodes formdata as form-urlencoded and sets the header for you. Crawlee's payload takes the raw request body, so encode the fields yourself with urllib.parse.urlencode and set the Content-Type through headers=. For a full login flow with session reuse, see the Logging in with a crawler guide.

Run on
import asyncio
from urllib.parse import urlencode

from crawlee import Request
from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    crawler = ParselCrawler(max_requests_per_crawl=10)

    @crawler.router.default_handler
    async def login_page(context: ParselCrawlingContext) -> None:
        # The CSRF token is tied to the session cookie issued for this GET.
        if not context.session:
            raise RuntimeError('Session not found')

        token = context.selector.css('input[name="csrf_token"]::attr(value)').get()

        # The CSRF token is required for the POST to succeed. If it's missing,
        # the login will fail.
        if not token:
            raise RuntimeError('CSRF token not found')

        form = {'csrf_token': token, 'username': 'user', 'password': 'pass'}

        # Crawlee's `payload` is the raw request body, so encode the fields yourself
        # and set the `Content-Type`. Scrapy's `FormRequest` does both for you.
        await context.add_requests(
            [
                Request.from_url(
                    'https://quotes.toscrape.com/login',
                    method='POST',
                    payload=urlencode(form),
                    headers={'content-type': 'application/x-www-form-urlencoded'},
                    label='after-login',
                    # Bind the POST to the same session so its CSRF cookie matches.
                    session_id=context.session.id,
                    # The POST shares the GET's URL. Include the method and payload
                    # in the unique key, or the queue drops it as a duplicate.
                    use_extended_unique_key=True,
                )
            ]
        )

    @crawler.router.handler('after-login')
    async def after_login(context: ParselCrawlingContext) -> None:
        logged_in = context.selector.css('a[href="/logout"]').get() is not None
        await context.push_data({'logged_in': logged_in})

    await crawler.run(['https://quotes.toscrape.com/login'])


if __name__ == '__main__':
    asyncio.run(main())

JavaScript rendering​

For pages that need a browser, Scrapy users add the scrapy-playwright package, which requires extra settings to register its download handlers and the asyncio reactor. Crawlee has browser support built in through the PlaywrightCrawler. The handler receives a Playwright Page on context.page, so you query the rendered DOM instead of a static response. For details, see the Playwright crawler guide.

Run on
import asyncio

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # `PlaywrightCrawler` renders JavaScript in a real browser. It replaces the
    # `scrapy-playwright` package, with browser support built into the framework.
    crawler = PlaywrightCrawler(
        headless=True,
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url}')

        # `context.page` is a Playwright `Page`. Query the rendered DOM directly.
        for quote in await context.page.locator('div.quote').all():
            await context.push_data(
                {
                    'text': await quote.locator('span.text').text_content(),
                    'author': await quote.locator('small.author').text_content(),
                }
            )

        await context.enqueue_links(selector='li.next a')

    await crawler.run(['https://quotes.toscrape.com/js/'])


if __name__ == '__main__':
    asyncio.run(main())

PlaywrightCrawler renders every request. In Scrapy you flag individual requests with meta={'playwright': True} and leave the rest on the plain downloader. For that mix in Crawlee, use AdaptivePlaywrightCrawler, which renders in a browser only when the page needs it. For details, see the Adaptive Playwright crawler guide.

Conclusion​

Moving from Scrapy to Crawlee is mostly a mechanical rewrite. The spider class becomes a crawler with handlers, callbacks become labels, and yield becomes push_data or enqueue_links. The Parsel selectors carry over untouched. In return you get browser rendering, proxy rotation, and storage under one API, on top of asyncio.

If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!