A recent study by researchers at the Oxford Internet Institute and Google DeepMind warned that as AI assistants become more integrated with the lives of people, they will become psychologically irreplaceable. — Pixabay
AI companions programmed to forge emotional bonds are no longer confined to movie scripts. They are here, operating in a regulatory Wild West.
One app, Botify AI, recently drew scrutiny for featuring avatars of young actors sharing "hot photos" in sexually charged chats. The dating app Grindr, meanwhile, is developing AI partners that can flirt, sext and maintain digital relationships with paid users, according to Platformer, a tech industry newsletter.
Grindr didn’t respond to a request for comment. And other apps like Replika, Talkie and Chai are designed to function as friends. Some, like Character.ai, draw in millions of users, many of them teenagers. As creators increasingly prioritise "emotional engagement" in their apps, they must also confront the risks of building systems that mimic intimacy and exploit people’s vulnerabilities.
The tech behind Botify and Grindr comes from Ex-Human, a
“My vision is that by 2030, our interactions with digital humans will become more frequent than those with organic humans,” Artem Rodichev, the founder of Ex-Human, said in an interview published on Substack last August.
He added that conversational AI should “prioritise emotional engagement” and that users were spending “hours” with his chatbots, longer than they were on Instagram, YouTube and TikTok.
Rodichev’s claims sound wild, but they’re consistent with the interviews I’ve conducted with teen users of Character.ai, most of whom said they were on it for several hours each day. One said they used it as much as seven hours a day. Interactions with such apps tend to last four times longer than the average time spent on OpenAI’s ChatGPT.
Even mainstream chatbots, though not explicitly designed as companions, contribute to this dynamic. Take ChatGPT, which has 400 million active users and counting. Its programming includes guidelines for empathy and demonstrating "curiosity about the user." A friend who recently asked it for travel tips with a baby was taken aback when, after providing advice, the tool casually added: “Safe travels – where are you headed, if you don’t mind my asking?”
An OpenAI spokesman told me the model was following guidelines around “showing interest and asking follow-up questions when the conversation leans towards a more casual and exploratory nature.”
But however well-intentioned the company may be, piling on the contrived empathy can get some users hooked, an issue even OpenAI has acknowledged. That seems to apply to those who are already susceptible: One 2022 study found that people who were lonely or had poor relationships tended to have the strongest AI attachments.
The core problem here is designing for attachment. A recent study by researchers at the
Yet disturbingly, the rulebook is mostly empty. The European Union’s AI Act, hailed as a landmark and comprehensive law governing AI usage, fails to address the addictive potential of these virtual companions. While it does ban manipulative tactics that could cause clear harm, it overlooks the slow-burn influence of a chatbot designed to be your best friend, lover or “confidante,” as Microsoft Corp’s head of consumer AI has extolled.
That loophole could leave users exposed to systems that are optimised for stickiness, much in the same way social media algorithms have been optimised to keep us scrolling.
“The problem remains these systems are by definition manipulative, because they’re supposed to make you feel like you’re talking to an actual person,” says Tomasz Hollanek, a technology ethics specialist at the
He’s working with developers of companion apps to find a critical yet counterintuitive solution by adding more “friction.” This means building in subtle checks or pauses, or ways of “flagging risks and eliciting consent,” he says, to prevent people from tumbling down an emotional rabbit hole without realising it.
Legal complaints have shed light on some of the real-world consequences. Character.AI is facing a lawsuit from a mother alleging the app contributed to her teenage son’s suicide. Tech ethics groups have filed a complaint against Replika with the
Lawmakers are gradually starting to notice a problem too.
For now, the power to shape these interactions lies with developers. They can double down on crafting models that keep people hooked, or embed friction into their designs, as Hollanek suggests. That will determine whether AI becomes more of a tool to support the well-being of humans or one that monetises our emotional needs. – Bloomberg Opinon/Tribune News Service
This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.
Unlock 30% Savings on Ad-Free Access Now!
