For companies that meet AIUC’s safety standards, they partner with some of the largest insurance companies to provide an insurance policy. — Photo by Possessed Photography on Unsplash
An early Anthropic employee has launched an AI insurance firm to help companies deploy artificial intelligence systems while mitigating the catastrophic risks of rogue AI models. The Artificial Intelligence Underwriting Company, which came out of stealth on Wednesday, was launched by co-founders Rune Kvist, formerly of Anthropic; Brandon Wang, a former Thiel fellow; and Rajiv Dattani, a former McKinsey insurance partner.
The trio raised US$15mil (RM63.5mil), primarily from former GitHub CEO Nat Friedman through his firm, NFDG. Friedman was responsible for US$10mil (RM42.3mil) of that investment, with the remaining third coming from Emergence, Terrain, and angel investors like Benjamin Mann of Anthropic.
A top-down approach to AI regulation, Kvist tells Inc., is too slow to keep up with how the technology is evolving. “The EU Act started drafting in 2021. So much has changed since then,” he says. Instead, AIUC is creating both the safety standards and underwriting AI insurance for companies that are implementing the technology.
“You wouldn’t rely on Goldman Sachs to say whether their bonds are worthwhile; you look to Standard & Poor,” he says. This independent perspective, Kvist says, is essential to accurately assessing the safety of AI models and their implementation.
For companies that meet AIUC’s safety standards, they partner with some of the largest insurance companies to provide an insurance policy. A retailer, Kvist says, may take out a policy to cover cases in which its AI chatbots go rogue and misinform users about its returns policies. An insurance-based approach, he adds, helps untangle one of the most pressing questions around AI safety: Who is responsible when the machine goes awry?
The team, which officially launched in December 2024 and had been operating in stealth until this week, currently consists of five people. But they intend to double headcount over the next year. “We’re trying to keep things tight,” Kvist says.
In a paper released earlier this month, Kvist, Dattani and Wang outlined their philosophy for how insurance can help hasten AI progress while managing the risks artificial intelligence poses to society.
“We’re navigating a tightrope as Superintelligence nears,” the trio wrote, referring to the advancement of artificial intelligence. “If the West slows down unilaterally, China could dominate the 21st century. If we accelerate recklessly, accidents will halt progress.”
AI regulation, however, according to the trio, is moving too slowly and is too much of a patchwork to be effective in mitigating the risks of the technology. Instead, they say “insurers are incentivised to develop and quickly iterate on core safety measures” and therefore could help lead AI regulation efforts.
However, AIUC isn’t an alternative to government regulation. Kvist still believes governments should play a key role in regulating larger-scale AI risks, like that posed by deepfakes, AI’s ability to create bioweapons, or other catastrophic risks.
“There’s no substitute to government regulation,” says Timnit Gebru, the founder of the Distributed AI Research Institute. “Insurance companies should be more attuned to the risks,” she adds, but Gebru worries that the rapid pace of AI adoption is at odds with reducing AI harms.
Kvist estimates enterprise adoption of AI is set to hit US$500bil (RM2.1 trillion) in the next five years. As it does, the market for insurance providers is set to grow exponentially. “We’re in uncharted territory without a parallel in human history,” he says. “We’re going to need to act with confidence around all the ambiguity and put numbers on the risks. The people you can trust to put numbers on it are those with skin in the game, who pay if we get the numbers wrong.” – Inc./Tribune News Service
