More than 1 in 5 U.S. companies now use artificial intelligence in their daily operations, according to a recent analysis of federal data from Goldman Sachs. In theory AI offers the promise of greater productivity and profits, but even with advances in capabilities, these systems are still fundamentally unpredictable, creating new liabilities for firms that use them. That’s where insurance typically comes in. But AI is disrupting that business, too. Deloitte projects insurance for AI will grow into a nearly $5 billion global business by 2032. One early mover is Corgi, a startup insurance company that uses AI and offers coverage for its mistakes. It also operates a buzzy cafe in San Francisco’s financial district.Unfortunately, you won’t find any corgi dogs at the cafe, thanks to city health codes. But you will find plenty of startup founders like Aimen Hallou, who’s been working from the cafe with his laptop and a couple buddies.“We have like five to six businesses, all very successful,” said Hallou. He wouldn’t go into much detail about what they all do exactly, lest someone steal his ideas, but broadly there’s a ski app, a sports betting analysis tool, a Reddit marketing service, a clipping service, and an internet proxy service — all powered by AI.“So it's like you do a lot of work in the beginning, and then you can kind of step away while it's making revenue,” said Hallou.CEO Nico Laqua said Corgi has built its business to serve this new generation of AI-native entrepreneurs — partially with protein shakes and espresso shots.“We don't have decaf for a reason,” he said. “I think people do need to lock in a little bit.”But the real business (and the namesake corgi) are in the office upstairs. The cafe space just came with the lease and turned into a fun side project.“We're best known for our insurance for technology companies,” Laqua said. That includes all the usual policies, like commercial general liability, errors and ommissions, and their new add-on for AI liability. “Just like a human might mess up, AI might mess up even worse,” Laqua said. “It might say things that it shouldn't say, it might calculate things it shouldn't calculate, it might even cause outages or things like that.”Corgi’s policy explicitly covers damages that could result from these AI mistakes. Josh Sirota bought the Corgi policy for his startup, Eragon, which builds agentic systems for businesses. It uses frontier AI models made by other companies, like OpenAI, Anthropic, or Google, and itself builds the scaffolding software that allows the models to act autonomously on business tasks.“Let's say that a business connected a bank account to agents that we've built,” said Sirota. “Even though we've tested these agents paying invoices hundreds of times, if there was a hallucination [on the part of the frontier model,] it would be our perspective that it wasn't due to any of the technology that we've built.”He wanted to be absolutely sure, and be able to give customers assurances, that any such problem would be covered. So he bought Corgi’s AI-specific insurance.Whether traditional business insurance policies cover such liabilities now is a bit untested, said John Farley, who leads the cyber practice at insurance brokerage Gallagher.“The insurance market is beginning to move from silence on AI to scrutiny of AI,” he said.Currently, most traditional policies simply don’t mention the technology because they predate it. But Farley said the insurance industry has begun standardizing language and seeking regulatory approvals to potentially exclude AI from their plans.Insurers may add these exclusions to future policies and spin off AI risk into a standalone add-on product, as they did for cybersecurity decades ago — and as Corgi and a handful of other plans are already doing.But defining what exactly counts as an AI-related loss could be a challenge, said Michael Levine, an attorney who represents policyholders with Hunton Andrews Kurth.“How do you put a box around it for purposes of a contract, in a way that's going to be clear and uniform?” he said, noting a lot of technology could be classified as AI that firms might not consider.“And then there's a whole other problem out there of what they're calling ‘shadow AI,’ which is the AI that that you don't know about,” he said. “It's the AI your your employees are downloading and using themselves.”But one of the biggest challenges AI poses for insurers is understanding what the risk looks like and how to price it.“Historically the insurance markets have used modeling tools to measure the likelihood of, say, a hurricane in Florida, right?” said Farley. “We've got lots of data there. Here, not so much.”How do you know if a brand new agentic AI system your company uses to process invoices is likely to make a costly mistake? That’s what Ines Boutemadja is trying to quantify with AI insurance startup Klaimee.“When we onboard a customer, what we actually do is we test their AI agents,” she said. “We try and actually trigger the potential errors that would lead to a claim,” like leaking sensitive customer data.In demo tests of agentic systems Klaimee ran through their evaluations, they have easily triggered failures. “We had the agent, in under like 10 minutes, give you information about the previous customer they talked with — their age, their personal information, banking details,” said Boutemadja.Agents that fail can be improved and tested again. The better the score, the lower the insurance premium. Several other AI insurance upstarts, like the AI Underwriting Company and Mount, are taking a similar approach.“We pride ourselves of being the enabler of technology adoption,” said Boutemadja. “There wouldn't be any skyscrapers in New York if you didn't have the proper insurance for it.”From building skyscrapers to empires of AI apps, innovation creates new risk. And for insurers – an opportunity to sell new products.That, and 24/7 coffee.