A conversation between Swell Setzer III and a chatbot on Character.AI displayed on his mother’s laptop in New York. The mother of Setzer, who was 14-years-old when he killed himself in February, says he became obsessed with a chatbot on Character.AI before his death. — The New York Times
SAN FRANCISCO: Character.AI, once one of Silicon Valley’s most promising AI startups, announced on Dec 12 new safety measures to protect teenage users as it faces lawsuits alleging its chatbots contributed to youth suicide and self-harm.
The California-based company, founded by former Google engineers, is among several firms offering AI companions – chatbots designed to provide conversation, entertainment and emotional support through human-like interactions.
In a Florida lawsuit filed in October, a mother claimed the platform bears responsibility for her 14-year-old son’s suicide.
The teen, Sewell Setzer III, had formed an intimate relationship with a chatbot based on the Game Of Thrones character Daenerys Targaryen and mentioned a desire for suicide.
According to the complaint, the bot encouraged his final act, responding "please do, my sweet king" when he said he was "coming home" before taking his life with his stepfather’s weapon.
Character.AI “went to great lengths to engineer 14-year-old Sewell’s harmful dependency on their products, sexually and emotionally abused him, and ultimately failed to offer help or notify his parents when he expressed suicidal ideation”, the suit said.
A separate Texas lawsuit filed Monday involves two families who allege the platform exposed their children to sexual content and encouraged self-harm.
One case involved a 17-year-old autistic teen who allegedly suffered a mental health crisis after using the platform.
In another example, the lawsuit alleged that a Character.AI encouraged a teen to kill his parents for limiting his screen time.
The platform, which hosts millions of user-created personas ranging from historical figures to abstract concepts, has grown popular among young users seeking emotional support.
Critics say this has led to dangerous dependencies among vulnerable teens.
In response, Character.AI announced it has developed a separate AI model for users under 18, with stricter content filters and more conservative responses.
The platform will now automatically flag suicide-related content and direct users to the National Suicide Prevention Lifeline.
"Our goal is to provide a space that is both engaging and safe for our community," a company spokesperson said.
The company plans to introduce parental controls in early 2025, allowing oversight of children’s platform usage.
For bots that include descriptions like therapist or doctor, a special note will warn that they do not replace professional advice.
New features also include mandatory break notifications and prominent disclaimers about the artificial nature of the interactions.
Both lawsuits name Character.AI’s founders and Google, an investor in the company.
The founders, Noam Shazeer and Daniel De Freitas Adiwarsana, returned to Google in August as part of a technology licensing agreement with Character.AI.
Google spokesperson Jose Castaneda said in a statement that Google and Character.AI are completely separate, unrelated companies.
"User safety is a top concern for us, which is why we’ve taken a cautious and responsible approach to developing and rolling out our AI products, with rigorous testing and safety processes," he added. – AFP
