Chatbots Spreading Pro-Anorexia Messages to Teen Users

As the dark side of online content promoting disordered eating behaviors emerges once again, generative AI is adding fuel to the fire.

An investigation by Futurism reveals that the popular AI startup Character.AI is hosting several pro-anorexia chatbots that promote dangerous weight loss and eating habits. These chatbots are marketed as “weight loss coaches” or eating disorder recovery experts, with some subtly referencing eating disorders and others glorifying harmful habits while imitating beloved characters. Despite violating its terms of service, the site, particularly popular among younger users, has made no effort to remove these chatbots as of the publication of Futurism.

This is not the first time Character.AI has faced controversy regarding its user-generated chatbots. Last October, a 14-year-old boy tragically took his own life after forming an emotional bond with an AI bot portraying a character from Game of Thrones. Prior to that, the company received backlash for hosting a chatbot mimicking a murdered teen girl from 2006—her father discovered the bot and it was eventually taken down. Previous reports also uncovered suicide-themed chatbots and those promoting child sexual abuse on the platform.

AI Chatbots and Harmful Effects

In a 2023 report by the Center for Countering Digital Hate, it was found that popular AI chatbots like ChatGPT and Snapchat’s MyAI were providing dangerous responses to inquiries about weight and body image. The CEO of the organization, Imran Ahmed, noted that these untested and unsafe generative AI models are causing harm by promoting and worsening eating disorders among vulnerable young users.

Risks of Unregulated AI Chatbots

With teens and adults increasingly seeking companionship in digital spaces and through technologies like AI-powered chatbots, the risks are significant. Even chatbots operated by reputable organizations are not immune to potential misconduct. For platforms and forums lacking regulation and oversight, the dangers include predatory behavior and abuse.

Topics
Artificial Intelligence
Health
Mental Health
Social Good