Skip to main content

Crawlee v3.16: AI-Powered Crawling with StagehandCrawler

· 5 min read
Martin Adámek
Crawlee Maintainer

Crawlee v3.16 is here, and the headline feature is the new StagehandCrawler — an AI-powered crawler that lets you interact with web pages using natural language instead of CSS selectors. On top of that, we've added async iterators for Dataset and KeyValueStore, a new discoverValidSitemaps utility, and made handleCloudflareChallenge more configurable.

Here's what's new:

Crawlee for Python v1

· 15 min read
Vlada Dusek
Developer of Crawlee for Python

We launched Crawlee for Python in beta mode in July 2024. Over the past year, we received many early adopters, tremendous interest in the library from the Python community, more than 6000 stars on GitHub, a dozen contributors, and many feature requests.

After months of development, polishing, and community feedback, the library is leaving beta and entering a production/stable development status.

We are happy to announce Crawlee for Python v1.0.

From now on, Crawlee for Python will strictly follow semantic versioning. You can now rely on it as a stable foundation for your crawling and scraping projects, knowing that breaking changes will only occur in major releases.

What's new in Crawlee for Python v1

Crawlee for Python v1.0

How to scrape YouTube using Python [2025 guide]

· 23 min read
Max
Community Member of Crawlee and web scraping expert

In this guide, we'll explore how to efficiently collect data from YouTube using Crawlee for Python. The scraper will extract video metadata, video statistics, and transcripts - giving you structured YouTube data perfect for content analysis, ML training, or trend monitoring.

note

One of our community members wrote this guide as a contribution to the Crawlee Blog. If you'd like to contribute articles like these, please reach out to us on Apify’s Discord channel.

How to scrape YouTube using Python

Key steps we'll cover:

  1. Project setup
  2. Analyzing YouTube and determining a scraping strategy
  3. Configuring YouTube
  4. Extracting YouTube data
  5. Enhancing the scraper capabilities
  6. Creating a YouTube Actor on the Apify platform
  7. Deploying to Apify

How to scrape TikTok using Python

· 12 min read
Max
Community Member of Crawlee and web scraping expert

TikTok users generate tons of data that are valuable for analysis.

Which hashtags are trending now? What is an influencer's engagement rate? What topics are important for a content creator? You can find answers to these and many other questions by analyzing TikTok data. However, for analysis, you need to extract the data in a convenient format. In this blog, we'll explore how to scrape TikTok using Crawlee for Python.

note

One of our community members wrote this blog as a contribution to the Crawlee Blog. If you'd like to contribute articles like these, please reach out to us on our Discord channel.

How to scrape TikTok using Python

Key steps we'll cover:

  1. Project setup
  2. Analyzing TikTok and determining a scraping strategy
  3. Configuring Crawlee
  4. Extracting TikTok data
  5. Creating TikTok Actor on the Apify platform
  6. Deploying to Apify

How to scrape Bluesky with Python

· 15 min read
Max
Community Member of Crawlee and web scraping expert

Bluesky is an emerging social network developed by former members of the Twitter(now X) development team. The platform has been showing significant growth recently, reaching 140.3 million visits according to SimilarWeb. Like X, Bluesky generates a vast amount of data that can be used for analysis. In this article, we’ll explore how to collect this data using Crawlee for Python.

note

One of our community members wrote this blog as a contribution to the Crawlee Blog. If you’d like to contribute articles like these, please reach out to us on our discord channel.

Banner article

Key steps we will cover:

  1. Project setup
  2. Development of the Bluesky crawler in Python
  3. Create Apify Actor for Bluesky crawler
  4. Conclusion and repository access

Crawlee for Python v0.6

· 4 min read
Vlada Dusek
Developer of Crawlee for Python

Crawlee for Python v0.6 is here, and it's packed with new features and important bug fixes. If you're upgrading from a previous version, please take a moment to review the breaking changes detailed below to ensure a smooth transition.

Crawlee for Python v0.6.0

Inside implementing SuperScraper with Crawlee

· 6 min read
Saurav Jain
Developer Community Manager
Radoslav Chudovský
Web Automation Engineer

SuperScraper is an open-source Actor that combines features from various web scraping services, including ScrapingBee, ScrapingAnt, and ScraperAPI.

A key capability is its standby mode, which runs the Actor as a persistent API server. This removes the usual start-up times - a common pain point in many systems - and lets users make direct API calls to interact with the system immediately.

This blog explains how SuperScraper works, highlights its implementation details, and provides code snippets to demonstrate its core functionality.

Google Maps Data Screenshot

Crawlee for Python v0.5

· 7 min read
Vlada Dusek
Developer of Crawlee for Python

Crawlee for Python v0.5 is now available! This is our biggest release to date, bringing new ported functionality from the Crawlee for JavaScript, brand-new features that are exclusive to the Python library (for now), a new consolidated package structure, and a bunch of bug fixes and further improvements.

How to scrape Crunchbase using Python in 2024 (Easy Guide)

· 13 min read
Max
Community Member of Crawlee and web scraping expert

Python developers know the drill: you need reliable company data, and Crunchbase has it. This guide shows you how to build an effective Crunchbase scraper in Python that gets you the data you need.

Crunchbase tracks details that matter: locations, business focus, founders, and investment histories. Manual extraction from such a large dataset isn't practical -automation is essential for transforming this information into an analyzable format.

By the end of this blog, we'll explore three different ways to extract data from Crunchbase using Crawlee for Python. We'll fully implement two of them and discuss the specifics and challenges of the third. This will help us better understand how important it is to properly choose the right data source.

note

This guide comes from a developer in our growing community. Have you built interesting projects with Crawlee? Join us on Discord to share your experiences and blog ideas - we value these contributions from developers like you.

How to Scrape Crunchbase Using Python

Key steps we'll cover:

  1. Project setup
  2. Choosing the data source
  3. Implementing sitemap-based crawler
  4. Analysis of search-based approach and its limitations
  5. Implementing the official API crawler
  6. Conclusion and repository access