Scraping One Million Bluesky Posts for AI Training!

Bluesky Faces First AI Scrape Despite User Data Assurance

Bluesky is already facing its first major AI scrape, despite the stance of its owners that it will never train generative AI on user data. Reported by 404Media on Nov. 26, one million public Bluesky posts — complete with identifying user information — were crawled and then uploaded to AI company Hugging Face. The dataset was created by machine learning librarian Daniel van Strien, intended to be used in the development of language models and natural language processing, as well as general analysis of social media trends, content moderation, and posting patterns. It contains users’ decentralized identifiers (DIDs) and even has a search function to find content from specific users.

Bluesky users didn’t opt-in to such uses of their content, but neither is it expressly prohibited by Bluesky. The platform’s firehose API is an “aggregated, chronological stream of all the public data updates as they happen in the network, including posts, likes, follows, handle changes, and more.” Bluesky’s API — coupled with the public and decentralized Authenticated Transfer (AT) Protocol the site is built on — means Bluesky content is open and available to the third party developers the platform is trying to court, 404Media explains.

Bluesky Rep Responds to Data Upload

A Bluesky representative responded to 404Media’s requests for comment: “Bluesky is an open and public social network, much like websites on the Internet itself. Just as robots.txt files don’t always prevent outside companies from crawling those sites, the same applies here. We’d like to find a way for Bluesky users to communicate to outside orgs/developers whether they consent to this and that outside orgs respect user consent, and we’re actively discussing how to achieve this.”

Shortly after the article’s publication, the dataset was removed from Hugging Face. “I’ve removed the Bluesky data from the repo. While I wanted to support tool development for the platform, I recognize this approach violated principles of transparency and consent in data collection. I apologize for this mistake,” van Strien wrote in a follow-up Bluesky post.

Topics: Artificial Intelligence, Social Good, Social Media.