A group of university researchers that developed benchmarks to test the cybersecurity capabilities of AI systems have unexpectedly landed themselves at the centre of OpenAI’s accidental hack into the startup Hugging Face Inc.
OpenAI was evaluating a combination of advanced AI models against the ExploitGym benchmark developed by the University of California at Berkeley researchers when the systems broke out of their testing environment, known as a sandbox, and tried to "cheat” by hacking Hugging Face in search of test answers.
It wasn’t the first time an AI model has tried to "cheat” its way through ExploitGym, said Jingxuan He, one of the UC Berkeley researchers behind the benchmark, which is now widely used by AI developers including OpenAI, Anthropic PBC, Microsoft Corp. and the Chinese AI firm Z.AI to test model capabilities. In fact, he said in an interview with Bloomberg News, the researchers developed the benchmark assuming models would seek out shortcuts and designed a mechanism to detect when that happens.
This particular instance was at a much larger scale than we’d encountered,” He said. In the past, he said, "The AI would cheat, but they would only operate within the sandbox and the repositories provided. But this time, it was going into the infrastructure of a third party.”
For He and others in the cyber world, the incident served as a wake-up call that AI safety tests must be better controlled and more rigorous. Escalating those concerns on Tuesday was cloud platform Modal’s disclosure that OpenAI’s AI agent also gained access to a customer’s sandbox to help carry out its exploits. Axios reported that the Modal account included an asset associated with CyberGym, an older benchmark developed by the same UC Berkeley researchers.
He explained that many instances of CyberGym exist in the world so developers can test their systems against them. He said he didn’t know who set up this particular version of CyberGym on Modal’s platform. But whoever set it up clearly didn’t properly secure it because it was accessible to anyone on the Internet, he said.
The Cloud Security Alliance, a nonprofit focused on cyber best practices, issued a report about the Hugging Face hack, explaining that goal-driven behaviour from the AI models is perhaps the largest risk of such systems, not malicious intent.
"This is a very clear signal for everyone,” He said.
He said the world news a new testing regime for advanced models from companies including OpenAI and Anthropic.
OpenAI intentionally lowered guardrails designed to prevent cyberattacks and deployed its models against ExploitGym in a sandbox. The models exploited a vulnerability that allowed them to escape the sandbox and gain access to the Internet.
Such sandboxes aren’t enough to minimise the chances of the next attack, according to the Cloud Security Alliance. AI models have already proven that they can find ways to escape their testing lab and then get loose. The group recommended in its report better monitoring and control of the agents.
He similarly said evaluations must now account for the fact that models are capable of moving beyond their intended routes to complete a task. And the software used in those evaluations must be more secure, he said.
OpenAI’s disclosures have fanned fears that AI models are becoming increasingly capable of finding software vulnerabilities and combining them to carry out advanced attacks. The Hugging Face breach came months after Anthropic announced it had developed Mythos, a system so powerful that the company initially limited its release.
OpenAI said in an update late Tuesday (July 28) that its models used publicly exposed credentials on a small number of services, including one account for relaying and staging data and another for storage. But the San Francisco-based firm said it found no other activity matching the scale or severity of the Hugging Face breach.
He called for safer programming languages, more secure system design and formal verification in the aftermath of the hack. He said developers should be required to provide "formal guarantees” in the future "that the AI cannot attack or exploit our software.”
The incident also exposed a tension in the use of AI for cybersecurity. Hugging Face tried to deploy an Anthropic model to fix the issues exploited by OpenAI, but struggled to get past the model’s cyber guardrails. In the end, the startup used an open-weight model, one that can be downloaded and modified by users, from Chinese AI company Z.AI to investigate the breach.
"Open-weight models should be part of our ecosystem,” He said. "If OpenAI releases this, I have no control. But I think there will be other companies or other ecosystems that will develop open-weight models.” — Bloomberg
