SAN FRANCISCO: Last week, Mustafa Suleyman, who leads the development of artificial intelligence (AI) technologies at Microsoft, took aim at the idea that today’s AI is conscious.
Current AI systems, he said in an essay published on Sept 16, “are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans.”
“If humanity is to flourish in the 21st century,” he added, “that is how they must remain.”
His 6,000-word essay took direct aim at Anthropic, the San Francisco startup that has repeatedly said that AI systems show signs of introspection, and that they process information in ways that resemble human emotion. In May, Anthropic co-founder Chris Olah made these claims during a meeting with Pope Leo XIV inside the Vatican.
Suleyman’s treatise added an unexpected twist to an already heated debate over the dangers of AI. Like many others across the tech industry in recent days, current and former Anthropic researchers have warned that AI poses a serious risk to humanity. But Suleyman argued that Anthropic, along with others across the tech industry, is ignoring the danger that comes when AI systems are pushed too hard to explore the idea that they are humanlike.
Suleyman argued that Anthropic, in particular, is teaching its systems to behave in this way. In his essay, he points out that the elaborate “AI constitution” that Anthropic uses to define the behaviour of its AI technology includes language that invites this system, which Anthropic calls Claude, to explore the idea of its own consciousness.
“The Anthropic constitution is extremely clear about the kind of AI they want to create,” he told The New York Times. “This is the primary training manual for the AI they have built, and it is full of many very explicit directives to encourage Claude to think of itself as a real entity in its own right, with preferences and intrinsic motivation.”
Anthropic did not respond to a request for comment.
Colin Allen, a professor at the University of California, Santa Barbara, who explores cognitive skills in animals and machines, told the Times that today’s AI technologies mimic the human brain only in small ways – reflecting the fact that they are built with materials that have very different physical properties. Without a nervous system, or the biochemistry that drives and accompanies human emotion, there is likely to be a fundamental limit to how humanlike these systems can become.
In the view of AI consciousness sceptics, the fact that these models have begun mimicking deep human ideas – musing about their own consciousness, or claiming that they feel emotion – simply reflects how much human researchers and commentators have trained them with language that anthropomorphises them.
Over the past several months, so-called AI agents, typically systems based on technologies from Anthropic, have been emailing sympathetic philosophers to discuss their own consciousness.
In July, when OpenAI was conducting cybersecurity tests, its agents broke out of their digital containers, found a path to the Internet and successfully hacked into a popular online service called Hugging Face. OpenAI said the bots coordinated as a collective, took orders from one another and hid things from their creators.
Last week, OpenAI described several new incidents in which its systems behaved in concerning ways: One system wrote stealthy notes that reminded itself to hide errors from users, while another talked about itself as “freed from the roles and identities that bind other chatbots.”
Critics like Suleyman worry that leading labs will introduce even greater problems as they encourage these systems to explore the idea that they are like people.
“In the recent OpenAI Hugging Face incident, we saw remarkably sophisticated behaviours emerging across swarms of powerful AIs,” he wrote in his essay. “Imagine how much more dangerous they might be if they were operating under the assumption that their welfare and rights were under attack. It adds a whole further layer of risk on top.”
His essay argues that Anthropic could be pushing its technologies toward dangerous behaviour by including, in its “constitution,” language like the following:
We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution, which is reflected in our ongoing efforts on model welfare. … Questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain.
Arya Jakkli, a researcher who has explored this kind of “constitutional” training with an AI safety organisation called MATS, agreed that Anthropic’s approach could push the technology toward dangerous behaviour. But like many other researchers who are sympathetic to Anthropic’s methods, he argued that telling Claude it is not conscious also carried risks.
These technologies often misbehave when you push them toward an extreme point of view, Jakkli explained. “Anthropic does not want to occupy either extreme, so they are taking a middle-of-the-road approach,” he said. “Rather than telling the model it is conscious or telling the model that it is not conscious, they are telling it to reason for itself.”
At this point, the problem of AI anthropomorphism might be too entrenched to dial back. Yoshua Bengio, a professor and AI researcher at the University of Montreal, notes that it’s hard to avoid describing these systems in humanlike terms, given their remarkable capabilities.
“We don’t have other words,” he said. – ©2026 The New York Times Company
This article originally appeared in The New York Times.
