Who Pays When AI Is Wrong?

The labs building frontier models are now publishing their own warnings. Read enough Twitter posts and you’ll find models that, in testing, have tried to blackmail operators, sandbag evaluations, or take actions nobody asked them to take. That creates an obvious question for any enterprise thinking about putting AI into a real workflow.

If the AI goes wrong, where does the liability fall?

Today, the uncomfortable answer is: it depends. Nobody really knows.

One early-stage company was recently asked to carry $4 million in insurance to close a $200,000 contract. No one involved could clearly explain which policy was meant to provide that coverage. That captures the state of AI liability today: buyers have already decided that AI represents a distinct risk. There is no consensus on how to define, price, or underwrite it.

Historically, when a large new risk emerges, someone eventually learns to price it. That is how insurance markets are created. Our view is that AI risk is still difficult to underwrite today - but over the next 12–24 months, AI insurance will emerge as a meaningful standalone category.

The coverage gap

Most software companies already buy some combination of tech E&O and cyber insurance.

Tech E&O is designed around a fairly intuitive idea: your product failed, you made a mistake, and someone lost money because of it. Cyber insurance serves a different problem: an external actor hacks a system through a breach, ransomware attack, or theft of data.

AI doesn't always look like either.

A support bot invents a refund policy. A claims model denies elderly patients at a higher rate. A voice agent reads another customer's account back to them. An image model generates something that turns out to infringe someone's copyright.

No attacker necessarily accessed the system. There may be no single negligent act or identifiable software defect. The system simply generated an outcome that caused harm.

Traditional policies were written for software that behaves predictably enough to investigate fault after the fact. AI systems must instead be assessed across a wide range of inputs and edge cases that may not have been seen before. 

This is what cyber looked like a decade ago

Cyber insurance went through something pretty similar.

For a long time, cyber risk was quietly sitting inside general liability and E&O. Coverage was unclear, exclusions were inconsistent, and brokers often reassured clients that they were “probably covered.”

Then the claims started coming in.

Global cyber premiums went from roughly $2B in 2015 to ~$7B in 2020 to ~$20B in 2025, with reinsurer estimates of around $60B by 2030. More than 220 insurer groups now write cyber directly, and roughly 360 MGAs do so through delegated authority in the US. The five largest cyber MGAs have raised more than $2B between them.

The hard days of cyber created the eventual winners. New underwriting models emerged, and companies such as Coalition helped turn cyber into a distinct insurance market.

AI will move faster for two reasons.

First, cyber risk is mostly about what happens to the system. AI risk potentially sits in every output the system generates. The exposure is continuous.

Second, regulation is arriving much earlier. Cyber took years to become a board-level issue. AI regulation is showing up almost immediately - the EU is already phasing in its rules, and Colorado's AI Act takes effect this year.

Tech E&O is roughly a $5B market (in premiums) inside a much larger professional liability market. Cyber is around $20B. It's not crazy to think AI liability could eventually be bigger than cyber and much more quickly.

The risk isn't theoretical anymore and conversations have started happening

There's the Air Canada chatbot case. Google's AI Overviews have already generated a nine-figure claim. There's a $1.5B copyright settlement. There is litigation around an insurer's claims-denial model.

We're also seeing it in enterprise procurement with MSAs having three-page AI risk controls. Security reports required after major model releases. Liability caps at 2 - 3x contract value.

So why isn't anyone really selling this? A few reasons:

First, underwriting is genuinely hard.

Cyber has a relatively nice underwriting setup. You can look at the perimeter. You can identify critical assets. You can run scans. You have a reasonably clear picture of the attack surface.

AI is much messier - One assessor we spoke to expected a company to be running a single model and found seven models chained together. Behaviour can change without the underlying product changing. And once a system is probabilistic, it's much harder to isolate what actually caused a particular failure.

There also isn't much historical loss data to work from - The underwriters we spoke to kept coming back to the same point: this is an insurance problem before it's a technology problem.

Second, AI is mostly silent cover today.

Most policies don't explicitly cover AI, but they don't explicitly exclude it either.

So everyone is operating in this grey area.

Fewer than a dozen carriers have explicit AI carve-outs. And when exclusions start appearing at the reinsurance layer, it can take 12–24 months for them to flow through the market.

Third, there haven't been enough big paid losses.

Cyber became a standalone market when insurers started writing real cheques. That hasn't really happened with AI yet.

There have been serious incidents. CrowdStrike is an obvious example. But incidents that happen without an insurer writing a material claim don't do much to change insurance behaviour. The AI equivalent of the big cyber losses hasn't arrived yet.

What the market looks like today

Carriers are cautious

There are a few carriers testing standalone AI products. A couple of specialty insurers will add AI by endorsement, often with sublimits around $250K. Most cyber MGAs are still putting AI inside existing cyber or E&O policies and underwriting it with questionnaires. Open-source models are also viewed as more difficult to insure because there isn't an obvious upstream party to transfer the risk to.

Nobody has the know-how

Underwriting talent is a constraint. The people who can price cyber are scarce; the people who can price a non-deterministic system don’t exist.

Demand is latent

Founders pay $3K - $20K a year for E&O plus cyber and told us they would pay $3 - 4K more for AI cover. But nearly every founder said the same thing: they will buy when a customer mandates it, not before - and most expect that within 12 months as agents move from information retrieval into transactions and refunds.

Insurance doesn't create demand here. It helps vendors get through procurement. That's basically what cyber insurance does for SaaS today.

What would actual AI underwriting look like?

What made the cyber MGAs work wasn't a nicer application form. It was a new data source.

Two hospitals on the same street with the same revenue and patient count look identical on paper; a scan of their security systems says they are not. Policy issuance changed from weeks to minutes.

The AI equivalent is adversarial testing of the live system. You run thousands of generated attacks - prompt injection, PII exfiltration, hallucination under pressure, discriminatory outputs - against the production agent and record the failure rate. 

‍

The second input is telemetry. 

Most AI companies already have traces flowing into observability systems. An insurer could potentially use that data to see whether production behaviour stays within the range that was tested. That opens up some interesting possibilities: repricing at renewal, pricing individual deployments, or flagging a new deployment that materially changes the risk.

‍

The third input is portfolio construction. 

A reinsurer doesn't just care whether one customer has a bad claim. It cares whether 500 customers can have the same bad claim at the same time. Imagine a major model provider pushes an update that creates a new failure situation. Or a prompt-injection technique gets discovered that works across a particular model family. Suddenly the same vulnerability is sitting inside hundreds of insured companies. That's the AI version of the systemic cyber losses seen in 2017–2020. Portfolio construction matters from day one. You probably don't want a book that's 40% dependent on one foundation model. You may want concentration limits by model and vertical. You may want to charge more for models that already represent too much of the book.

‍

After actuarial discipline, the core of the business depends on distribution.

Technology alone will not build this category. More than 98% of US commercial insurance is sold through brokers, and brokers need a clear reason to introduce a new product to their clients.

The companies that win this market will have strong technology for underwriting, carrier relationships and actuarial discipline. It is a hard, multi-year build. It is also how a multi-billion-dollar business got made in cyber insurance.