Alphabet Inc’s Google has begun rolling out Gemini 4 Argon, its long-awaited flagship artificial intelligence model, but the company is grappling with internal scepticism over how well it performs in key areas, such as coding.
Google unveiled the model Wednesday to a small group of trusted cybersecurity partners and said it will expand access after additional testing, starting with paid subscribers. The company said Gemini 4 posted leading scores on several benchmark tests, including beating OpenAI’s Astra model on one that measures security skills.
But some insiders say those metrics don’t tell the whole story. While Gemini 4 has performed well on benchmarks widely used to gauge model efficacy, it does less well when employees actually put it to work, according to people with direct access to the effort. The model struggles to handle certain coding tasks, said the people, who requested anonymity to discuss an internal matter.
Alphabet shares rose about 1.7% in extended trading after the company’s announcement. Late in regular trading, they pared Wednesday’s earlier gains after Bloomberg reported employee scepticism about the model.
Google has laboured recently to develop models that can compete with OpenAI and Anthropic PBC. The company had planned to release a different version in June, dubbed Gemini 3.5 Pro, but abandoned the effort, the people said.
Google said it would be inaccurate to say that Gemini 4 is underperforming in areas such as coding. The company referred Bloomberg back to comments made last week by Koray Kavukcuoglu, the head of Google DeepMind who said he was encouraged by the model’s performance.
"I have the utmost trust in the team,” Kavukcuoglu said at a conference hosted by tech news site The Information. "In my mind, it’s a certainty that we are always gonna be at the frontier.”
There is a spectrum of opinion inside Google. Some employees believe Anthropic’s Fable and OpenAI’s Astra models are improving at a faster rate than Gemini. These people believe that Gemini 4 – even at its best – will still lag behind those models in some areas. Other employees believe the coming version has caught up with the leading AI labs.
A Google employee familiar with model development said there is "large consensus” internally at the company that Gemini 4 is at the frontier. This person said the company had conducted rigorous tests of the models and denied that they struggle with messy, real-world coding tasks.
Google badly needs Gemini 4 to succeed. Versions of the model underpin nearly every product the company sells, from the AI answers atop Search, Google’s main profit engine, to Maps, Gmail and Chrome. Each of those products has more than a billion users, a distribution advantage some of the company’s rivals lack.
But OpenAI and Anthropic are increasingly moving beyond selling models to building products of their own, including coding agents. Failing to deliver a cutting-edge model could give Google’s competitors more time to convince consumers, developers and businesses that the future of search and software should run on their platforms instead.
In response to questions from Bloomberg, Google said that even though its last Pro model was released in February, the company has since seen growth in its AI products, including the enterprise version of Gemini as well as its consumer chatbot app and AI Mode in Google Search, the last two of which have crossed 1 billion users.
Gemini 3
Last November, Google debuted Gemini 3, a well-received model that was widely seen as a turning point for the company’s efforts to keep up with OpenAI and Anthropic. Google announced a new iteration called Gemini 3.5 Pro at its I/O conference in May and pledged to release the model the following month. But that deadline passed, and the company has since abandoned 3.5 Pro, according to people familiar with the matter.
Besides hampering Google’s AI ambitions, the decision likely cost the company dearly in time and money. Training runs to build such a model can require spending as much as US$400mil (RM1.6bil), according to Bloomberg Intelligence analyst Mandeep Singh. Using highly paid AI researchers could boost the price tag.
Now Google is encountering various challenges with Gemini 4. Its coding abilities are uneven, according to people familiar with the model’s internal evaluations. Gemini isn’t particularly adept at front-end design, which shapes how apps and websites look and feel, one person said. That’s a potentially serious setback because Google has struggled to compete in the red-hot market for AI coding tools.
Moreover, it’s a very large model, according to a person familiar with its development. Typically big models are expensive to run, potentially putting pressure on Google’s margins.
‘Benchmaxxing’
Experts say Google could be suffering from an industry tendency to focus on benchmarks – a phenomenon known as "benchmaxxing,” when engineers concentrate more on achieving a good score than creating a product that does a job well. AI labs tend to do this because customers often judge models by their benchmark scores. Gemini 4 appears to be affected by this process, said two people familiar with the model.
Edwin Chen, the founder of AI startup Surge AI, said relying on benchmarks can prompt labs to focus on building models that write code in a particular language, rather than creating apps that are easy to use or well-designed.
"An analogy would be, ‘Oh yeah, my kid got a really good score on the SAT’ - but the SAT doesn’t translate into real-world performance,” Chen said. "It’s an incredibly pernicious problem.”
Gemini 4 does have strengths, according to a person familiar with the matter, who said the model stands out at making sense of inputs beyond text, such as extracting metadata from video. They also pointed to the model’s safety, cyber security and ability to communicate clearly and naturally.
The frustration inside Google is palpable. Researchers interviewed for a previous Bloomberg story blamed a sprawling bureaucracy that’s trying to weave the technology into nearly everything Google makes, and said changing mandates and shifting priorities have made it difficult to focus on a cohesive strategy.
Meanwhile, a wave of star AI researchers have left Google, including legendary engineer Jeff Dean, Nobel Prize-winner John Jumper and Noam Shazeer, who helped invent the technology that underpins much of the AI boom. In August, Demis Hassabis, who has long led the company’s AI research, stepped into a new role as chairman and ceded day-to-day operations at DeepMind to Kavukcuoglu, a longtime lieutenant.
Google said on Wednesday that Gemini 4 is built for long, complex jobs in software engineering, finance, legal and cybersecurity work. It can produce longer responses than its predecessor, Google said – up to 1 million tokens, or roughly 750,000 words, in one go.
As the company works to catch up, rivals continue to hurtle forward despite talk of slowing development of some frontier models after a series of incidents in which AI agents hacked into outside organisations. Earlier this month, Meta Platforms Inc. released Muse, an AI agent that it said can complete everyday tasks like shopping online and booking appointments. The app quickly zoomed to the top of the download charts. – Bloomberg
