👉 This article was written by Shaheer Sarfaraz as part of Write for Apify - a program for developers sharing original articles about what they've built with Crawlee.

I spent three days debugging a scraper that I really expected to have knocked out in an afternoon. At one point I had four different AI models open simultaneously, all confidently giving me solutions that did absolutely nothing. I fixed it eventually. By thinking. Like an animal.

What JobOps does​

JobOps is a self-hosted job search pipeline I built because I was tired of manually checking job boards that quietly delete listings the moment they expire. It scrapes eight international and UK-specific job sources (like LinkedIn, Indeed), normalizes the results, and stores them so I can search historical data instead of whatever happens to be live today. It's got over 2,700 stars on GitHub, which I think means other people had the same problem.

Most of the sources were relatively easy to add. A couple took an afternoon. One took the better part of a week and is still the scraper I trust least. That one is Gradcracker, the main UK graduate job board for engineering and tech, and the source that taught me more about Crawlee than everything else in JobOps combined.

This is that story.

Crawlee v3.16 is here, and the headline feature is the new StagehandCrawler — an AI-powered crawler that lets you interact with web pages using natural language instead of CSS selectors. On top of that, we've added async iterators for Dataset and KeyValueStore, a new discoverValidSitemaps utility, and made handleCloudflareChallenge more configurable.

Here's what's new:

• StagehandCrawler — AI-powered browser automation
• Async iterators for Dataset and KeyValueStore
• discoverValidSitemaps utility
• Improved Cloudflare challenge handling

We launched Crawlee for Python in beta mode in July 2024. Over the past year, we received many early adopters, tremendous interest in the library from the Python community, more than 6000 stars on GitHub, a dozen contributors, and many feature requests.

After months of development, polishing, and community feedback, the library is leaving beta and entering a production/stable development status.

We are happy to announce Crawlee for Python v1.0.

From now on, Crawlee for Python will strictly follow semantic versioning. You can now rely on it as a stable foundation for your crawling and scraping projects, knowing that breaking changes will only occur in major releases.

What's new in Crawlee for Python v1​

• New storage client system
• Adaptive Playwright crawler
• Impit HTTP client
• Sitemap request loader
• Robots exclusion standard
• Fingerprinting
• Open telemetry

In this guide, we'll explore how to efficiently collect data from YouTube using Crawlee for Python. The scraper will extract video metadata, video statistics, and transcripts - giving you structured YouTube data perfect for content analysis, ML training, or trend monitoring.

Key steps we'll cover:

1. Project setup
2. Analyzing YouTube and determining a scraping strategy
3. Configuring YouTube
4. Extracting YouTube data
5. Enhancing the scraper capabilities
6. Creating a YouTube Actor on the Apify platform
7. Deploying to Apify

TikTok users generate tons of data that are valuable for analysis.

Which hashtags are trending now? What is an influencer's engagement rate? What topics are important for a content creator? You can find answers to these and many other questions by analyzing TikTok data. However, for analysis, you need to extract the data in a convenient format. In this blog, we'll explore how to scrape TikTok using Crawlee for Python.

Key steps we'll cover:

1. Project setup
2. Analyzing TikTok and determining a scraping strategy
3. Configuring Crawlee
4. Extracting TikTok data
5. Creating TikTok Actor on the Apify platform
6. Deploying to Apify

Build a price tracker with Crawlee for Python to scrape product details, export data in multiple formats, and send email alerts for price drops, then deploy and schedule it as an Apify Actor.

Bluesky is an emerging social network developed by former members of the Twitter(now X) development team. The platform has been showing significant growth recently, reaching 140.3 million visits according to SimilarWeb. Like X, Bluesky generates a vast amount of data that can be used for analysis. In this article, we’ll explore how to collect this data using Crawlee for Python.

Key steps we will cover:

1. Project setup
2. Development of the Bluesky crawler in Python
3. Create Apify Actor for Bluesky crawler
4. Conclusion and repository access

Crawlee for Python v0.6 is here, and it's packed with new features and important bug fixes. If you're upgrading from a previous version, please take a moment to review the breaking changes detailed below to ensure a smooth transition.

SuperScraper is an open-source Actor that combines features from various web scraping services, including ScrapingBee, ScrapingAnt, and ScraperAPI.

A key capability is its standby mode, which runs the Actor as a persistent API server. This removes the usual start-up times - a common pain point in many systems - and lets users make direct API calls to interact with the system immediately.

This blog explains how SuperScraper works, highlights its implementation details, and provides code snippets to demonstrate its core functionality.

Crawlee for Python v0.5 is now available! This is our biggest release to date, bringing new ported functionality from the Crawlee for JavaScript, brand-new features that are exclusive to the Python library (for now), a new consolidated package structure, and a bunch of bug fixes and further improvements.