Skip to main content
Version: Next

Migrating from Scrapy

Scrapy and Crawlee solve the same problem, so most of what you know carries over. The concepts line up almost one to one, and the selector code often moves across without changes. Scrapy parses HTML with Parsel, and so does Crawlee's ParselCrawler. Your response.css() and response.xpath() queries keep working on context.selector.

This guide maps Scrapy concepts to their Crawlee equivalents and rewrites a small spider step by step. The examples use ParselCrawler because it's the closest match for a Scrapy spider. The other HTTP option is BeautifulSoupCrawler, and the choice between them is covered in the HTTP crawlers guide. For pages that need a browser, Crawlee has PlaywrightCrawler, covered in the Playwright crawler guide.

The Crawlee examples cap each run with max_requests_per_crawl to keep the demos short. The Scrapy equivalent is the CLOSESPIDER_PAGECOUNT setting from the built-in CloseSpider extension.

Installation​

Install Crawlee with the Parsel extra:

pip install 'crawlee[parsel]'

For the browser examples, also install Playwright:

pip install 'crawlee[playwright]'
playwright install

Conceptual differences​

Both frameworks give you a request scheduler, a duplicate filter, retries, and a way to pull data out of pages. The way you wire those pieces together differs.

  • Scrapy runs on the Twisted reactor, and your callbacks are synchronous generators that yield items and requests. Crawlee runs on asyncio, and handlers are coroutines defined with async def that await helpers like push_data and enqueue_links. You write straight-line async/await code without a reactor.
  • Scrapy starts spiders through its own scrapy crawl command, so tracing the program flow and debugging it step by step takes extra setup. A Crawlee project is a regular Python script. You run it with python main.py and step through it in any debugger.
  • A Scrapy spider is a class with a parse method and callbacks. Crawlee routes requests through a Router that dispatches them to handler functions by their label.
  • Browser rendering, proxy rotation, session management, and storage are part of the framework. In Scrapy you reach for scrapy-playwright, proxy middlewares, and feed exporters. In Crawlee these are built in and share one API.

Mapping key concepts​

ScrapyCrawlee
scrapy.SpiderParselCrawler
start_urls / start_requests()crawler.run([...])
parse(self, response)@crawler.router.default_handler
callback=self.parse_detail@crawler.router.handler('detail')
cb_kwargs / Request.metaRequest.user_data
response.css() / response.xpath()context.selector
yield {...}push_data
scrapy.ItemA plain dict
response.follow() / yield Request(...)enqueue_links / add_requests
dont_filter=TrueRequest.from_url(always_enqueue=True)
allowed_domainsenqueue_links(strategy=...)
scrapy.FormRequestRequest.from_url(method='POST', payload=...)
Item pipelinesDataset
Downloader / spider middlewaresrouter.use(), navigation hooks, HTTP clients
settings.pyConfiguration + crawler arguments
CONCURRENT_REQUESTS / DOWNLOAD_DELAYConcurrencySettings
AutoThrottleThrottlingRequestManager
DEPTH_LIMITmax_crawl_depth
CLOSESPIDER_PAGECOUNTmax_requests_per_crawl
DOWNLOAD_TIMEOUTnavigation_timeout
ROBOTSTXT_OBEYrespect_robots_txt_file
RetryMiddlewaremax_request_retries
errbackfailed_request_handler
FEEDS / feed exportsexport_data
CrawlSpider / Rulesenqueue_links(selector=..., label=...)
HttpProxyMiddlewareProxyConfiguration
scrapy-playwrightPlaywrightCrawler
scrapy crawl myspiderpython main.py

Migrating a spider​

Here's the canonical quotes spider from the Scrapy tutorial. It reads quotes from a listing page and follows the pagination link. The Crawlee version keeps the selectors untouched. The parse method becomes the default handler, yield {...} becomes push_data, and the manual pagination becomes a single enqueue_links call.

import scrapy


class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

Note that:

  • context.selector is the Parsel Selector that Scrapy calls response. Every CSS and XPath query works the same way.
  • enqueue_links extracts the href from matching elements, resolves it against the current URL, and schedules it. It also skips URLs already seen, so you don't reimplement the duplicate filter.
  • push_data writes to the default Dataset. There's no separate item pipeline to declare.

Routing with labels​

Scrapy routes pages by passing a callback to each request. A listing page hands its links to parse_author. In Crawlee you attach a label to the enqueued requests, then register a handler for that label. Each request goes to the matching handler, and unlabeled requests fall through to the default one. For details, see the Request router guide.

import scrapy


class AuthorsSpider(scrapy.Spider):
    name = 'authors'
    start_urls = ['https://quotes.toscrape.com/']

    def parse(self, response):
        for href in response.css('div.quote span a::attr(href)').getall():
            yield response.follow(href, callback=self.parse_author)

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

    def parse_author(self, response):
        yield {
            'name': response.css('h3.author-title::text').get(),
            'born': response.css('span.author-born-date::text').get(),
            'bio': response.css('div.author-description::text').get(),
        }

Passing data between handlers​

A listing page often holds data you want to keep, then merge with fields from the detail page. Scrapy carries it through cb_kwargs (or Request.meta). Crawlee carries it through Request.user_data, which travels with the request and is available on context.request.user_data in the next handler.

How you attach it depends on the data. When each link carries its own value, like a price that belongs to one book, create a request per URL with Request.from_url(..., user_data=...) and add them with add_requests. That's what the Crawlee example does. When a single value applies to the whole batch, skip the loop and pass user_data straight to enqueue_links.

import scrapy


class BooksSpider(scrapy.Spider):
    name = 'books'
    start_urls = ['https://books.toscrape.com/']

    def parse(self, response):
        for book in response.css('article.product_pod'):
            href = book.css('h3 a::attr(href)').get()
            # Carry the listing price into the detail callback via `cb_kwargs`.
            yield response.follow(
                href,
                callback=self.parse_book,
                cb_kwargs={'listing_price': book.css('p.price_color::text').get()},
            )

    def parse_book(self, response, listing_price):
        yield {
            'title': response.css('h1::text').get(),
            'listing_price': listing_price,
        }

Rule-based crawling​

Scrapy's CrawlSpider declares Rules with a LinkExtractor to follow links automatically and dispatch them to callbacks. Crawlee expresses the same intent with enqueue_links. The restrict_css and restrict_xpaths arguments map to selector, and the allow and deny patterns map to include and exclude, which accept Glob patterns or regular expressions.

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BooksSpider(CrawlSpider):
    name = 'books'
    start_urls = ['https://books.toscrape.com/']

    rules = (
        # Follow pagination, no callback.
        Rule(LinkExtractor(restrict_css='li.next')),
        # Extract each book detail page.
        Rule(
            LinkExtractor(restrict_css='article.product_pod h3', allow=r'/catalogue/'),
            callback='parse_book',
        ),
    )

    def parse_book(self, response):
        yield {
            'title': response.css('h1::text').get(),
            'price': response.css('p.price_color::text').get(),
        }

Exporting data​

Scrapy writes results through the FEEDS setting or the -O command-line flag. Crawlee collects items in the default Dataset as you call push_data, then export_data writes the whole dataset once the run finishes. The format is chosen by the file extension, either .json or .csv.

# settings.py
FEEDS = {
    'quotes.json': {'format': 'json'},
    'quotes.csv': {'format': 'csv'},
}

The default (unnamed) Dataset stays on disk after the run finishes, but the next run clears it before starting, since Configuration.purge_on_start defaults to True. To keep data across runs, use a named dataset or turn that option off. For details, see the Storages guide. To continue an interrupted crawl from where it stopped, see the Resuming a paused crawl example.

Concurrency and throttling​

Scrapy tunes throughput through several settings. Crawlee gathers the fixed controls into ConcurrencySettings and scales concurrency automatically based on system resources, as described in the Scaling crawlers guide. Use max_tasks_per_minute to cap the request rate. It's close to DOWNLOAD_DELAY, but it limits the whole pool rather than adding a fixed delay per domain.

# settings.py
CONCURRENT_REQUESTS = 20
DOWNLOAD_DELAY = 0.5

The closest built-in analog to the AutoThrottle extension is ThrottlingRequestManager, covered in the Request throttling guide. It differs in what drives the backoff. AutoThrottle adapts to measured latency, while the manager reacts to explicit signals: HTTP 429 replies and robots.txt crawl-delay directives. It wraps a RequestQueue and applies the backoff per domain. You list the domains to watch, and the crawler reports the signals to the manager on its own.

Crawl-delay works only when the crawler fetches robots.txt. A project generated by scrapy startproject obeys robots.txt, but Crawlee doesn't by default. Pass respect_robots_txt_file=True to keep that behavior. The flag enforces the Disallow rules on its own. It reads crawl-delay only together with ThrottlingRequestManager, and without the manager the crawler logs a warning.

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

Proxies​

Scrapy rotates proxies through a middleware or a third-party package. Crawlee takes a ProxyConfiguration and rotates the URLs for you. For tiered proxies, session-bound proxies, and integration with the Apify Proxy, see the Proxy management guide.

# settings.py
# Rotation needs the third-party `scrapy-rotating-proxies` package. The built-in
# `HttpProxyMiddleware` reads a single proxy from `request.meta` or the environment.
ROTATING_PROXY_LIST = [
    'http://proxy-1.com/',
    'http://proxy-2.com/',
]

DOWNLOADER_MIDDLEWARES = {
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}

Error handling​

Scrapy retries failed requests with RetryMiddleware and reports terminal failures through errback. Crawlee retries with max_request_retries, runs an error_handler between attempts, and calls a failed_request_handler once the retries run out. For details, see the Error handling guide.

import scrapy


class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com/']

    def start_requests(self):
        for url in self.start_urls:
            # `RetryMiddleware` retries the request automatically before `errback` fires.
            yield scrapy.Request(url, callback=self.parse, errback=self.on_error)

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {'text': quote.css('span.text::text').get()}

    def on_error(self, failure):
        self.logger.error('Request failed: %s', failure.request.url)

Forms and login​

Scrapy submits forms with FormRequest, which encodes formdata as form-urlencoded and sets the header for you. Crawlee's payload takes the raw request body, so encode the fields yourself with urllib.parse.urlencode and set the Content-Type through headers=. For a full login flow with session reuse, see the Logging in with a crawler guide.

import scrapy


class LoginSpider(scrapy.Spider):
    name = 'login'
    start_urls = ['https://quotes.toscrape.com/login']

    def parse(self, response):
        # `from_response` picks up the hidden `csrf_token` field on its own, encodes
        # the data as `form-urlencoded`, and sets the `Content-Type` header.
        yield scrapy.FormRequest.from_response(
            response,
            formdata={'username': 'user', 'password': 'pass'},
            callback=self.after_login,
        )

    def after_login(self, response):
        yield {'logged_in': bool(response.css('a[href="/logout"]'))}

JavaScript rendering​

For pages that need a browser, Scrapy users add the scrapy-playwright package, which requires extra settings to register its download handlers and the asyncio reactor. Crawlee has browser support built in through PlaywrightCrawler, with nothing to register. The handler receives a Playwright Page on context.page, so you query the rendered DOM instead of a static response. For details, see the Playwright crawler guide.

settings.py
# Without these, `meta={'playwright': True}` is silently ignored and the page is
# fetched by the default downloader with no rendering.
DOWNLOAD_HANDLERS = {
    'http': 'scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler',
    'https': 'scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler',
}
TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'
quotes_js.py
import scrapy


class JsQuotesSpider(scrapy.Spider):
    name = 'js-quotes'

    def start_requests(self):
        yield scrapy.Request(
            'https://quotes.toscrape.com/js/',
            meta={'playwright': True},
        )

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            # Every followed request has to opt back into rendering.
            yield response.follow(
                next_page,
                callback=self.parse,
                meta={'playwright': True},
            )

PlaywrightCrawler renders every request. In Scrapy you flag individual requests with meta={'playwright': True} and leave the rest on the plain downloader. For that mix in Crawlee, use AdaptivePlaywrightCrawler, which renders in a browser only when a page needs it. For details, see the Adaptive Playwright crawler guide.

Conclusion​

Moving from Scrapy to Crawlee is mostly a mechanical rewrite. The spider class becomes a crawler with handlers, callbacks become labels, and yield becomes push_data or enqueue_links. The Parsel selectors carry over untouched. In return you get browser rendering, proxy rotation, and storage under one API, on top of asyncio.

If you have questions or need assistance, feel free to reach out on our GitHub or join our Discord community. Happy scraping!