Skip to main content

Apify platform

Apify is a platform built to serve large-scale and high-performance web scraping and automation needs. It provides easy access to compute instances (Actors), convenient request and result storages, proxies, scheduling, webhooks and more, accessible through a web interface or an API.

While we think that the Apify platform is super cool, and it's definitely worth signing up for a free account, Crawlee is and will always be open source, runnable locally or on any cloud infrastructure.

note

We do not test Crawlee in other cloud environments such as Lambda or on specific architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.

Logging into Apify platform from Crawlee​

To access your Apify account from Crawlee, you must provide credentials - your API token. You can do that either by utilizing Apify CLI or with environment variables.

Once you provide credentials to your Apify CLI installation, you will be able to use all the Apify platform features, such as calling Actors, saving to cloud storages, using Apify proxies, setting up webhooks and so on.

Log in with CLI​

Apify CLI allows you to log in to your Apify account on your computer. If you then run your crawler using the CLI, your credentials will automatically be added.

npm install -g apify-cli
apify login -t YOUR_API_TOKEN

Log in with environment variables​

Alternatively, you can always provide credentials to your Actor by setting the APIFY_TOKEN environment variable to your API token.

There's also the APIFY_PROXY_PASSWORD environment variable. Actor automatically infers that from your token, but it can be useful when you need to access proxies from a different account than your token represents.

Log in with Configuration​

Another option is to use the Configuration instance and set your api token there.

from apify import Actor

sdk = Actor(Configuration(token='your_apify_token'));

What is an Actor​

When you deploy your script to the Apify platform, it becomes an Actor. An Actor is a serverless microservice that accepts an input and produces an output. It can run for a few seconds, hours or even infinitely. An Actor can perform anything from a simple action such as filling out a web form or sending an email, to complex operations such as crawling an entire website and removing duplicates from a large dataset.

Actors can be shared in the Apify Store so that other people can use them. But don't worry, if you share your Actor in the store and somebody uses it, it runs under their account, not yours.

Related links

Running an Actor locally​

First let's create a boilerplate of the new Actor. You could use Apify CLI and just run:

apify create my-hello-world

The CLI will prompt you to select a project boilerplate template - let's pick "Crawlee + BeautifulSoup". The tool will create a directory called my-hello-world with Python project files. You can run the Actor as follows:

cd my-hello-world
apify run

Running Crawlee code as an Actor​

For running Crawlee code as an Actor on Apify platform you need to wrap the body of the main function of your crawler with async with Actor.

NOTE

Adding async with Actor is the only important thing needed to run it on Apify platform as an Actor. It is needed to initialize your Actor (e.g. to set the correct storage implementation) and to correctly handle exitting the process.

Let's look at the BeautifulSoupCrawler example from the Quick Start guide:

import asyncio

from apify import Actor

from crawlee import Glob
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext


async def main() -> None:
    async with Actor:
        crawler = BeautifulSoupCrawler()

        @crawler.router.default_handler
        async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
            url = context.request.url

            # Extract HTML title of the page.
            title_element = context.soup.find('title')
            title = title_element.text if title_element else ''
            context.log.info(f'Title of {url}: {title}')

            # Add URLs that match the provided pattern.
            await context.enqueue_links(include=[Glob('https://www.iana.org/*')])

            # Save extracted data to dataset.
            await context.push_data({'url': url, 'title': title})

        # Enqueue the initial request and run the crawler
        await crawler.run(['https://www.iana.org/'])


asyncio.run(main())

Note that you could also run your Actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:

apify run

Deploying an Actor to Apify platform​

Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:

apify push

Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the Apify Actor documentation.

Usage on Apify platform​

You can also develop your Actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the Actors page in the app, click Create new and then go to the Source tab and start writing the code or paste one of the examples from the Examples section.

Storages​

There are several things worth mentioning here.

Helper functions for default Key-Value Store and Dataset​

To simplify access to the default storages, instead of using the helper functions of respective storage classes, you could use:

Using platform storage in a local Actor​

When you plan to use the platform storage while developing and running your Actor locally, you should use Actor.open_key_value_store(), Actor.open_dataset() and Actor.open_request_queue() to open the respective storage.

Using each of these methods allows to pass the force_cloud keyword argument. If set to True, cloud storage will be used instead of the folder on the local disk.

note

If you don't plan to force usage of the platform storages when running the Actor locally, there is no need to use the Actor class for it. The Crawlee variants KeyValueStore.open(), Dataset.open() and RequestQueue.open() will work the same.

Exporting dataset data​

When the Dataset is stored on the Apify platform, you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the Actor run details page and in the Storage section in the Apify Console. The actual data is exported using the Get dataset items Apify API endpoint. This way you can easily share the crawling results.

Related links

Environment variables​

The following describes select environment variables set by the Apify platform. For a complete list, see the Environment variables section in the Apify platform documentation.

note

It's important to notice that CRAWLEE_ environment variables don't need to be replaced with equivalent APIFY_ ones. Likewise, Crawlee understands APIFY_ environment variables.

APIFY_TOKEN​

The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage or to run an Actor on the Apify platform. You can find your API token on the Account Settings / Integrations page.

Combinations of APIFY_TOKEN and CRAWLEE_STORAGE_DIR​

By combining the env vars in various ways, you can greatly influence the Actor's behavior.

Env VarsAPIStorages
none OR CRAWLEE_STORAGE_DIRnolocal
APIFY_TOKENyesApify platform
APIFY_TOKEN AND CRAWLEE_STORAGE_DIRyeslocal + platform

When using both APIFY_TOKEN and CRAWLEE_STORAGE_DIR, you can use all the Apify platform features and your data will be stored locally by default. If you want to access platform storages, you can use the force_cloud=true option in their respective functions.

APIFY_PROXY_PASSWORD​

Optional password to Apify Proxy for IP address rotation. Assuming Apify Account was already created, you can find the password on the Proxy page in the Apify Console. The password is automatically inferred using the APIFY_TOKEN env var, so in most cases, you don't need to touch it. You should use it when, for some reason, you need access to Apify Proxy, but not access to Apify API, or when you need access to proxy from a different account than your token represents.

Proxy management​

In addition to your own proxy servers and proxy servers acquired from third-party providers used together with Crawlee, you can also rely on Apify Proxy for your scraping needs.

Apify Proxy​

If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be logged in to your Apify account.

from apify import Actor

proxy_configuration = await Actor.create_proxy_configuration()
proxy_url = await proxy_configuration.new_url();

Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create ProxyConfiguration instances. For using the Apify Proxy you should create an instance using the Actor.create_proxy_configuration() function instead.

Apify Proxy Configuration​

With Apify Proxy, you can select specific proxy groups to use, or countries to connect from. This allows you to get better proxy performance after some initial research.

import { Actor } from 'apify';

proxy_configuration = await Actor.create_proxy_configuration(
    groups=['RESIDENTIAL'],
    country_code='US',
);
proxy_url = await proxy_configuration.new_url();

Now your crawlers will use only Residential proxies from the US. Note that you must first get access to a proxy group before you are able to use it. You can check proxy groups available to you in the proxy dashboard.

Apify Proxy vs. Own proxies​

The ProxyConfiguration class covers both Apify Proxy and custom proxy URLs so that you can easily switch between proxy providers. However, some features of the class are available only to Apify Proxy users, mainly because Apify Proxy is what one would call a super-proxy. It's not a single proxy server, but an API endpoint that allows connection through millions of different IP addresses. So the class essentially has two modes: Apify Proxy or Own (third party) proxy.

The difference is easy to remember.

  • If you're using your own proxies - you should create a ProxyConfiguration instance directly.
  • If you are planning to use Apify Proxy - you should create an instance using the Actor.create_proxy_configuration() function. The new_url_function parameter enables the use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.

Related links