Skip to main content

Setting up

To run Crawlee on your computer, ensure you meet the following requirements:

  1. Python 3.9 or higher installed,
  2. Pip installed.

You can verify these by running the following commands:

python --version
pip --version

Installation​

Crawlee is available as the crawlee PyPI package. To install the core package, use:

pip install crawlee

After installation, verify that Crawlee is installed correctly by checking its version:

python -c 'import crawlee; print(crawlee.__version__)'

Crawlee offers several optional features through package extras. You can choose to install only the dependencies you need or install everything if you don't mind the package size.

Install all features​

If you do not care about the package size, install Crawlee with all features:

pip install 'crawlee[all]'

Installing only specific extras​

Depending on your use case, you may want to install specific extras to enable additional functionality:

BeautifulSoup​

For using the BeautifulSoupCrawler, install the beautifulsoup extra:

pip install 'crawlee[beautifulsoup]'

Parsel​

For using the ParselCrawler, install the parsel extra:

pip install 'crawlee[parsel]'

Curl impersonate​

For using the CurlImpersonateHttpClient, install the curl-impersonate extra:

pip install 'crawlee[curl-impersonate]'

Playwright​

If you plan to use a (headless) browser with PlaywrightCrawler, install Crawlee with the playwright extra:

pip install 'crawlee[playwright]'

After installing the playwright extra, install the necessary Playwright dependencies:

playwright install

Installing multiple extras​

You can install multiple extras at once by using a comma as a separator:

pip install 'crawlee[beautifulsoup,curl-impersonate]'

With Crawlee CLI​

The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have Pipx installed:

pipx --help

Then, run the CLI and choose from the available templates:

pipx run crawlee create my-crawler

If you already have crawlee installed, you can spin it up by running:

crawlee create my-crawler

Next steps​

Next, you will learn how to create a very simple crawler and Crawlee components while building it.