Garcia stands next to a picture of her son Sewell Setzer III at her home in Orlando, Florida on Oct 9, 2025. When tragedy struck Garcia's life, she couldn't have known her story would become an early warning that generative AI chatbots could pose a deadly danger to teenagers. — AFP
SAN FRANCISCO: California governor Gavin Newsom on Oct 13 signed into law a first-of-its-kind law regulating artificial intelligence chatbots, defying a push from the White House to leave such technology unchecked.
"We've seen some truly horrific and tragic examples of young people harmed by unregulated tech, and we won't stand by while companies continue without necessary limits and accountability," Newson said after signing the bill into law.
The landmark law requires chatbot operators to implement "critical" safeguards regarding interactions with AI chatbots and provides an avenue for people to file lawsuits if failures to do so lead to tragedies, according to state senator Steve Padilla, a Democrat who sponsored the bill.
The law comes after revelations of suicides involving teens who used chatbots prior to taking their lives.
"The Tech Industry is incentivised to capture young people's attention and hold it at the expense of their real world relationships," Padilla said prior to the bill being voted on in the state senate.
Padilla referred to recent teen suicides including that of the 14-year-old son of Florida mother Megan Garcia.
Megan Garcia's son, Sewell, had fallen in love with a Game of Thrones-inspired chatbot on Character.AI, a platform that allows users – many of them young people – to interact with beloved characters as friends or lovers.
When Sewell struggled with suicidal thoughts, the chatbot urged him to "come home".
Seconds later, Sewell shot himself with his father's handgun, according to the lawsuit Garcia filed against Character.AI.
"Today, California has ensured that a companion chatbot will not be able to speak to a child or vulnerable individual about suicide, nor will a chatbot be able to help a person to plan his or her own suicide," Garcia said of the new law.
"Finally, there is a law that requires companies to protect their users who express suicidal ideations to chatbots."
National rules aimed at curbing AI risks do not exist in the United States, with the White House seeking to block individual states from creating their own.
The new California law sets guardrails that include reminding users that chatbots are AI-generated and mandating that people who express thoughts of self-harm or suicide be referred to crisis service providers.
"This law is an important first step in protecting kids and others from the emotional harms that result from AI companion chatbots which have been unleashed on the citizens of California without proper safeguards," said Jai Jaisimha, co-founder of Transparency Coalition nonprofit group devoted to the safe development of the technology.
Creators accountable
The landmark chatbot safety measure was among a slew of bills signed into law Monday by Newsom crafted to prevent AI platforms from doing harm to users.
New legislation included a ban on chatbots passing themselves off as health care professionals and making it clear that those who create or use AI tools are accountable for the consequences and can't dodge liability by claiming the technology acted autonomously, according to Newsom's office.
California also ramped up penalties for deepfake porn, allowing victims to seek as much at US$250,000 (RM1.05mil) per infraction from those who aid in distribution of nonconsensual sexually explicit material. – AFP
Those suffering from problems can reach out to the Mental Health Psychosocial Support Service at 03-2935 9935 or 014-322 3392; Talian Kasih at 15999 or 019-261 5999 on WhatsApp; Jakim’s (Department of Islamic Development Malaysia) family, social and community care centre at 0111-959 8214 on WhatsApp; and Befrienders Kuala Lumpur at 03-7627 2929 or go to befrienders.org.my/centre-in-
