Setting up
To run Crawlee on your computer, ensure you meet the following requirements:
You can verify these by running the following commands:
python --version
pip --version
Installationโ
Crawlee is available as the crawlee
PyPI package. To install the core package, use:
pip install crawlee
After installation, verify that Crawlee is installed correctly by checking its version:
python -c 'import crawlee; print(crawlee.__version__)'
Crawlee offers several optional features through package extras. You can choose to install only the dependencies you need or install everything if you don't mind the package size.
Install all featuresโ
If you do not care about the package size, install Crawlee with all features:
pip install 'crawlee[all]'
Installing only specific extrasโ
Depending on your use case, you may want to install specific extras to enable additional functionality:
BeautifulSoupโ
For using the BeautifulSoupCrawler
, install the beautifulsoup
extra:
pip install 'crawlee[beautifulsoup]'
Parselโ
For using the ParselCrawler
, install the parsel
extra:
pip install 'crawlee[parsel]'
Curl impersonateโ
For using the CurlImpersonateHttpClient
, install the curl-impersonate
extra:
pip install 'crawlee[curl-impersonate]'
Playwrightโ
If you plan to use a (headless) browser with PlaywrightCrawler
, install Crawlee with the playwright
extra:
pip install 'crawlee[playwright]'
After installing the playwright extra, install the necessary Playwright dependencies:
playwright install
Installing multiple extrasโ
You can install multiple extras at once by using a comma as a separator:
pip install 'crawlee[beautifulsoup,curl-impersonate]'
With Crawlee CLIโ
The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have Pipx installed:
pipx --help
Then, run the CLI and choose from the available templates:
pipx run crawlee create my-crawler
If you already have crawlee
installed, you can spin it up by running:
crawlee create my-crawler
Next stepsโ
Next, you will learn how to create a very simple crawler and Crawlee components while building it.