Red Teams Still Land 25% of Attacks. Insurance Is the Brake.

Nathan Labenz opens with a bet he liked enough to put his own money in. Rune Kvist and Rajiv Dattani, co-founders of the AI Underwriting Company, want to certify and insure AI agents so enterprises can adopt them without pretending the downside is theoretical. Labenz joined the seed round alongside Nat Friedman, Emergence, and Terrain. The core claim, which the guests have used as a company introduction, is that security and progress are not a tradeoff. Get the first wrong and you may not get the second.

Before anyone reaches extinction scenarios, Kvist points at today's more mundane pileup: data leakage, safety and security failures, brand disasters, old harms made worse because there are more vulnerabilities and people understand them less. Scale that forward and the systems get gigantic and much smarter. Terrorists with access to brilliant scientific help could, in this telling, spin up new viruses more cheaply. Misalignment that looks like slight deception today could, as models get smarter, start to look like the frauds a high-IQ CEO can run. National-security leaders who believe AI confers a decisive strategic advantage might even attack first, in a war that looks conventional and is really about the model.

Labenz's shorthand for the bad future is the nuclear outcome: humanity did not get the super-cheap energy nuclear technology should have delivered, and it did get thousands of weapons hanging over everyone. He worries AI could be weaponized or locked into a cold war while the day-to-day quality-of-life gains get frozen by a few early incidents. Airplanes are the contrast he prefers: dramatic explosions are possible, yet people fly cheaply, and the ride to the airport is still the more dangerous part. The end state of a transformative technology, he argues, can hinge on a handful of early accidents. Kvist says some of that split is already visible. You still cannot drive an autonomous vehicle very far outside San Francisco before you get stopped. Drone warfare is increasingly autonomous, with no similar brake in sight.

The positive vision is a trinity: insurance, standards, and audits. Kvist's analogy is a race driver. The helmet and seatbelt are there so you can take corners faster. The better you can steer and secure AI systems, the faster you can deploy them and lean into agentic capabilities. The doom-versus-acceleration debate, in this frame, misses a flywheel. Dattani walks the mechanics. Insurance is who pays when something goes wrong. A standard is the codified best practice. An audit asks whether you actually met it. Insurers cannot tell a good risk from a bad one without clear requirements, the way reckless driving changes a car premium. A standard without an audit is self-attestation, and self-attestation is a trust gap.

Historically, insurers funded the standards and the auditors, then created the reason for risk-creating companies to comply: they wanted coverage, and cheaper coverage. Capital on insurance balance sheets paid for research; the people creating the risk wanted to stay insurable. Dattani sees that as a middle path between two poles that have taken over AI governance talk. One pole says let developers cook on voluntary commitments. The other wants strict top-down rules yesterday. Insurers want you to grow, because that is how premiums grow, so they will not load the product with onerous rules that nobody will buy. They also will not want the rules too lax, because they are the ones who pay. That, he says, is a market mechanism that can track both progress and security.

The first time the flywheel showed up in American practice, they argue, was 1752. Philadelphia's population grew tenfold in a decade, houses packed closer, fire risk spread, and no single homeowner carried the whole-city downside. Benjamin Franklin's fire insurance company had to pay when a house burned, so it wrote building codes, how far apart houses should stand, whether they could sit under large trees that lightning might hit, and then sent the first inspectors into homes to see whether the codes were met. Standards, audits, insurance. In the 1900s, electricity set houses on fire again, and insurers funded Underwriters Laboratories. The UL mark on a light bulb is a product-safety standard they then audit against. That is the mechanism they want to bring to AI.

Labenz's homework, via ChatGPT, was that the *kinds* of AI harms already sit inside existing policies, with a haze around whether an AI cause knocks an incident out of scope. Kvist and Dattani say the state of play is ambiguity. AI can create new cyber vulnerabilities; people already buy cyber insurance; it is unclear whether AI-induced holes are covered, because the word AI is not in the policy. They expect a split like the early 2000s, when computers made confidential leaks and downtime happen at a different frequency and severity. Lawsuits dragged on for years, insurers sued customers and vice versa, and cyber was carved into its own products because it had to be priced and structured differently. One thing they claim to know for sure: current policies have not priced AI risk, because insurers do not know how.

Historical loss tables will not save them. AI moves faster than anything insurers have underwritten. A product built today on historical AI data would be a pre-LLM product, blind to agentic risk. They also lack a way to collect the data. An EY study, Dattani says, finds enterprises taking million-dollar AI losses; smaller companies sit in the thousands to tens of thousands. Insurers may not even see those events. A long tail of small losses is being written off inside companies, while the press notices the big ones. Air Canada's chatbot inventing a refund policy, a passenger relying on it, and the company paying out is the kind of headline that finally asks a court whose mistake an AI mistake is. Without a product, there is no data; without data, there is no product.

Two tools are meant to break that loop. Red teaming and evals, familiar to the AI crowd, become a pool of synthetic data for insurers: this failed in these ways; if that happened in the world, here is the loss from a data leak or a lawsuit, which is their bread and butter. Frequency and severity from the tests plug into a pricing model. The other tool is parametric triggers. Instead of years of litigation over fault, the parties pre-agree that a verifiable out-of-distribution behavior, a line crossed on a specified test, pays a set amount. Fast, and cheaper than a court. Kvist's other point is easy to miss: even with no policy, someone is underwriting. A hospital or bank that deploys an AI system is already pricing the go or no-go. Labenz puts it in one line. We are all self-insured, whether we know it or not. The insurance industry's rigor, they argue, is more likely to get that price right than a non-expert in every enterprise making the call alone.

Labenz worries the loss curve looks more like venture returns or lab leaks than car crashes: a power law, a few events carrying most of the weight, one COVID out of many viruses that went nowhere. Nobody knows the tail, Kvist says, because nobody has seen it yet. Nuclear plant insurance is the analogy they trust. U.S. law requires every operator to carry insurance and to take liability no matter where in the supply chain the fault sits. Liability is capped: damage above $15 billion comes off the government balance sheet. The market may not be able to carry the largest tails. The government still wants an insurance industry, because it supplies independent third-party auditors with a direct financial reason to price risk and enforce practice, something government may not have the skill or the speed to do. Small, frequent losses, in this telling, are friends. They produce the data that lets you climb toward the rare, expensive ones. Nation-scale disaster still needs a public backstop. The private layer has to exist first.

After the break they walk through AIUC-1, which Labenz calls the first Western standard he knows of for building and deploying an AI agent. The design goal is to gather the concerns that slow adoption into one framework, detailed enough that an independent auditor can check whether a company is doing the work. They met more than 500 people inside the enterprises actually deploying these systems: security leaders, general counsel, banking, health, retail. The question was what keeps you up at night. The answers will not shock anyone. Data leakage at a new scale. Incorrect advice to customers. Jailbreaks and prompt injections. Discrimination against applicants. Old problems, new frequency and blast radius.

In practice, a bank looking at a customer-support chatbot asks for the AIUC-1 report: data and privacy controls, security implementations, output filtering, bias and role-tracking, whether the bot answers anger with anger or handles a refund demand correctly, groundedness and hallucination mitigation, human-in-the-loop practice, societal risks. One third-party document is supposed to preempt the questionnaire pile. Labenz asks how you boil 500 people into a consensus, and where the line was between overkill and a gap. Most of it, they found, was uncontroversial. People wanted a shared language. The standard often specifies what you must disclose, not a single required bar, because a hospital and a retailer have different risk appetites. The report puts information on the table; the buyer decides whether they can live with it.

The live debate sits in societal harm. Should you check whether a product makes large-scale cyberattacks easier, a question that may feel remote to any one buyer? They made judgment calls about what serves adoption for the industry, not just the account, because one incident, as with nuclear, can chill a whole sector. They still want the check to be pragmatic enough that you are not spending a fortune asking whether every random coding tool could take down an industry. Keep tabs, they say, without making the audit unserious.

Demand, they say, comes from both sides. Enterprises are nervous, ask endless questions, and sometimes simply do not buy. Developers are stuck earning trust one blog post at a time, grading their own homework, or floating financial guarantees their balance sheets cannot carry. The most forward-looking builders want the confidence infrastructure as much as the buyers. Labenz had asked Tasklet founder Andrew Lee, whose bet is to skip flowcharts and let models figure it out, whether he would buy insurance. Lee said the agents had not gone rogue yet, and he still wanted a policy he could walk into an enterprise deal with. That is the thesis in a customer quote.

Impact is supposed to show up in numbers, not slogans. When they certify a developer they run evals, often in multiple rounds. In the first round they tend to find a lot the company itself wants to fix. Attack success can reach 25% on certain classes of attack, a number no security team wants to show a bank. Then the standard is used to add a groundedness filter, monitoring, a content-moderation layer. The same failure rates, they say, can drop by 90% by the second or third round. There is no panacea. AI security is still an open research problem. Basic steps, done and retested, produce a curve a vendor can take to a customer: we were a mess, three months later the attacks mostly bounce. That, Kvist argues, is what makes a hungry company spend engineering time when a million other things are on the backlog.

Labenz asks whether the model stops at application developers wrapping APIs, or climbs into foundation models and the physical data-center layer. Dattani says it applies at all three, and is most acute at applications now: thinner balance sheets, thicker promises, and they are the ones actually shipping agents. A chair of audit and risk at a top tech company told them the firm is pouring an astounding amount into data centers, and public shareholders do not want to hold the downtime, attack, or breach risk. Can we buy insurance, and can a third party assure us those risks are managed in a way they were not for yesterday's facilities? AIUC-1 does not cover that. They expect to rerun the 500-executive process with data-center buyers and builders, codify practice, then insure off it. Foundation models get the same treatment later.

Kvist describes it as a ladder. Start with risks that cost millions or tens of millions, then hundreds of millions or low billions, then tens of billions. Adjacent places with many small incidents give you data and the right to climb. Labenz's field sketch: data-center operators at least know they are self-insuring tens of billions in the ground. Foundation-model labs, with notable exceptions, taxonomize risks and build defense in depth. The application layer, for most of the last two years, was weekend hackathons that became businesses with zero protections. That is the low-hanging fruit, and the place enterprise buyers most need a way to tell a best-effort shop from a launch-and-pray one.

Kvist mostly agrees, with color. Buy Claude.ai and the promise is a helpful tool that will hallucinate, not correct medical advice 90% of the time. Buy a customer-support agent and the public claim is we will resolve X% of tickets, and, implied, we will not create a brand disaster while talking to your customers. Thicker promises need thicker insurance. The line between foundation models and agentic apps is also blurring. Claude Code is not a foundation model; it is a system of models plus integrations and tool access. Agents, they argue, is the more useful frame. Application companies got there first. The labs are moving fast into the same space. Demand is two-way: a bank that builds its own internal support agent still wants a third party to say it is safe.

An audit starts in a database of real incidents. Air Canada's chatbot hallucinated a refund policy: check for financially loaded hallucinations. A paper showing chatbots that try to blackmail a user under experimental conditions goes in as a more speculative incident. From that pile they abstract a taxonomy of harms and attack vectors, the latest jailbreak research, the prompt-playing threads on Twitter, and turn it into a checklist. For each harm they specify technical, legal, and operational safeguards, plus the evals you must run. A groundedness filter to cut hallucinations is technical. An AI incident plan for when things hit the fan is operational. They look in the codebase and they read the policy manuals. Then they red-team, because a filter that does not work is not a filter. The report shows what is in place and how the tests landed, and leaves the go or no-go with the enterprise.

Labenz has just read *The Attacker Moves Second*, from Nicholas Carlini, Sander Schulhoff, and others. Static defenses beat the attacks you tuned them against; human red teamers then find a way around, in that study at a 100% clip. How do you avoid shoe-removal security theater, and how do you know when to stop? There is no neat answer, Kvist says. They run several rounds per audit, and certified companies commit to quarterly retests. The product will change, the incident world will change, new papers will land, and those get folded into the body of tests. Cybersecurity already lives here: with enough effort every system is reachable. Intelligence getting cheaper may lower the cost of a good audit even as the attack surface grows. Where you stop is a market call. What level of assurance will people pay for? Messy, and how every other domain actually settles it.

The company building the system pays, yearly, for a one-year certificate that requires quarterly technical tests. That can be an AI-native vendor or an enterprise rolling its own, a bank that thinks an in-house support agent is a moat and still wants confidence before it goes live. Price scales with risk surface and with value: bigger companies generate more revenue from the same badge and usually present a bigger surface. The same buyers often want insurance on top, to take risk off their customers' table or to deploy faster themselves. They are experimenting with certifying not just a product but a specific deployment, a bank's voice-support bot against that bank's policies and customer set, the extra layer that unlocks a contract because it is run in the actual environment.

Before a contract they do a gap analysis: the standard, outside-in public evidence, then a row-by-row review of where the company stands. That yields an effort estimate, engineering and non-engineering, and a price. Labenz's five-to-six-figure yearly range is, they say, broadly right. The gap analysis can land within 24 hours. Their largest customer has thousands of employees and needed a small handful of hours from the security team, mostly evidence collection. Their smallest, two or three people when they started, needed hands-on help implementing controls and remediating test findings. Recertification at year-end is meant to feel like SOC 2 or ISO, not a novel ritual.

Does every AI app eventually need this, the way every driver needs insurance? Dattani splits the question. Almost every valuable use case that ingests customer data or private IP needs someone who is not marking their own homework. Insurance may bifurcate. Traditional SaaS often sold on a have-a-nice-day basis, service credits if you are lucky. Cars and houses are mandated. High-risk deployments that create externalities should sit on a real balance sheet. Labenz's own company, Waymark, making local-business commercials, might reasonably decide the risk is manageable without a policy. Third-party testing, in this split, is closer to universal than payouts are.

Labenz spent enough time in financial services to remember the rating agencies. By the late 2000s they were captured: rate us high or we walk next door. Highly rated bonds went bad. How do you keep audits weird enough, and how do you stop a race to the bottom once competitors exist? Kvist's answer is insurance as skin in the game. Moody's had reputation and no payout mechanism, so it did not internalize the risk it blessed. UL, by contrast, was created by insurers who had the loss data and a direct reason to make the standard cover what was going wrong. That is the history they want to copy.

Keeping up with the frontier is a quarterly grind they do not pretend their own team can finish. They pull three kinds of input: the agent builders they are certifying, who know which new safeguard is actually working; academic and commercial safeguard people from places they name including Stanford, the University of Illinois, and Haize Labs; and an enterprise consortium of Fortune 500 security and risk leaders on the front line of deployments. Those people flag what AIUC-1 is missing and help write requirements that capture the spirit. On the insurance side they will not sit on their own balance sheet. They plan to become a managing general agent. Insurers pay them, and their pay is tied to underwriting profit. Large losses mean they get paid less or not at all. If losses stay ugly, the carriers walk and the company dies. They want partners who have paid claims for hundreds of years rather than a giant book of their own. Insurance, Dattani notes, is a cost-of-capital and risk-aggregation game. They want to stay focused on AI risk.

Labenz asks about California's SB 813, an opt-in to private standards in exchange for liability protection, and about liability reform more broadly. Dattani's test is simple. Anything that grows a market for oversight, that incentivizes robust evaluation without crowding out startups, is directionally right. What SB 813 did not do, in their view, is force the companies creating the risk to own it. Too many free passes, too many safe harbors, and you can manufacture risk without being held to account. Insurance, here, is how you internalize the externality. You can transfer it, but you pay for it. Risk does not vanish because a policy exists. If a framework lets the people who create risk walk away clean, it will fail over time. That is a design problem, not a vibes problem.

Early traction spans modalities. On the startup side they work with Ada and Intercom in customer support, Cognition on code, Recraft on image generation. The enterprise consortium includes risk and security leaders such as people thinking about bank-wide AI at JPMorgan Chase, the CISO of Confluent, and Anthropic's deputy CISO, plus healthcare and retail, and voices from Asia, Australia, and Europe. Labenz closes the way he opened: a private-sector reliability stack, powered by customers who want to move at all responsible speed, seems more likely to get the details right more than once as the technology changes. The constraint they are betting on is not model quality. It is whether trust can be made verifiable, priced, and sold.