From Character AI to a 1% Warning Shot: Gabriel Weil's Case for Liability as AI Governance

Nathan Labenz opens a short series on creative AI governance by arguing that governments face a singularly hard problem. Capabilities and adoption are moving fast, the timing and impact of AGI remain radically uncertain, and international competition makes heavy-handed rules look costly. Companies racing to one-up one another, he says, cannot simply be trusted to do the right thing for everyone else. The question is whether any tool can keep up with the technology, cut the most important risks, and still leave room for the upside — including, in his phrase, the dream of curing all diseases.

Gabriel Weil, assistant professor of law at Touro University and senior fellow at the Institute for Law and AI, offers an old instrument for that new problem. Rather than predict failures from a distance and prescribe safety standards, he wants liability law to make developers account for the risk their training and deployment choices impose on the rest of society. Labenz notes two structural advantages. The liability a company faces scales naturally with the risks it takes; if the systems are safe, there is little to worry about. And unlike most other proposals, the doctrine is already on the books and has adapted, over centuries, to new technologies.

Weil is less interested in rewriting how labs serve customers with AI doctors or self-driving cars, and more interested in harm to third parties who never sat inside the commercial relationship. Those people include pedestrians who share the road with robotaxis and the public facing some added pandemic or systemic risk. Inside that category he treats *misuse* — a person trying to cause harm — quite differently from *misalignment*, where the system itself "breaks bad." His most provocative move is to use punitive damages so that a small, payable injury can still bill the company for a catastrophe that almost happened.

Weil starts with a liability primer aimed at engineers. Negligence requires five elements: a duty of care, a breach of that duty, failure to exercise reasonable care, causation in fact and proximately, and an actual physical injury — not purely emotional harm. In an AI case, a plaintiff would have to show some alignment technique or safety practice that a reasonable person would have used, that the company skipped it, and that using it would have prevented this injury. In practice, he says, the breach inquiry is much narrower than a full social-cost accounting.

A driver who hits a pedestrian is not asked whether the trip's value justified the risk to people on the street, or whether an SUV's extra protection was worth the extra danger to others. Weil expects the same pattern in AI. Courts will not ask whether it was reasonable to train and deploy a high-capability system given the current state of alignment science. They will ask whether some off-the-shelf technique would have prevented this injury. That will generate liability in some cases. It will not, he argues, be an adequate standard while technical safety problems remain unsolved.

Products liability looks stricter and often is not. Software is typically treated as a service, so a purely digital model may never enter the regime; free models and one-off fine-tunes for a single customer sit outside it too. Manufacturing defects come closest to genuine strict liability, but AI almost never "ships the wrong weights." Warning defects collapse into disclaimers. Design defects collapse into a risk-utility balance that looks a lot like negligence. If no available design would have prevented the injury given the state of alignment science, Weil does not expect liability for failing to have solved the science.

Two more doctrines are speculative but relevant. Vicarious liability makes a principal pay for an agent's torts, classically an employer for an employee; AI systems are not legal persons, so the theory needs a vessel of liability that does not yet exist. Abnormally dangerous activities — blasting with dynamite, crop-dusting, keeping a pet tiger — impose liability even when reasonable care is used. Weil thinks that if judges understood frontier risk as he does, treating training and deployment as abnormally dangerous would not be a major doctrinal leap. He also thinks it will look weird to them, by default, to analogize a subset of software development to keeping a tiger. Compensatory damages make the plaintiff whole; they fail when the harm is too large to collect. That is why he reaches for punitive damages: if a small, payable injury looks like it could easily have become an uninsurable catastrophe, the company should pay for the unrealized risk, because that is the only case in which the legal system can still reach it.

Labenz asks the obvious policy question: why not just write rules? Weil first recasts AI risk as an externality. Training and deploying systems with unpredictable capabilities and uncontrollable goals imposes harm on third parties who are neither builders nor customers and who cannot opt out. Standard economics says those activities will be oversupplied unless the cost is priced. Climate policy can use a carbon tax because a given activity's contribution to risk is measurable *ex ante* and a particular hurricane is almost impossible to attribute *ex post*. AI, he argues, is the reverse: it is hard to tax a model's contribution to total risk up front, and relatively easier to attribute a harm after it happens. Liability is therefore well positioned.

Ex ante regulation also collides with orders-of-magnitude disagreement. Weil cites Eliezer Yudkowsky, who treats human extinction as nearly certain, against Marc Andreessen and a16z's Martin Casado, who treat the same risks as negligible. Stringent approval rules require society to pay large upfront costs, which requires consensus those costs are worth it. Even cheap measures — basic model testing, transparency, information preservation — have been hard to enact. Telling companies how to train, what safeguards to ship, and when they may deploy will be harder still for people who do not take the risks seriously.

Liability does not require that consensus. If you think misalignment is a rounding error, you should not fear being sued when an alignment failure happens. If the risks are large, the bill scales with them. In theory, Weil says, everyone can agree that you should pay for the harm you cause. The other advantage is expertise: cost-effective mitigations live inside the frontier labs, and he is pessimistic about moving that knowledge into government on government salaries. He would rather shift the onus onto companies to keep looking for new ways to make systems safer than write down a licensing regime that pretends to guarantee adequate safety at reasonable cost.

The mindset he wants is more precise than "anything that goes wrong is on you." He distinguishes alignment failures, capability failures, and misuse. For third-party harms from misalignment — the system did something the user did not want, by means the user would reject, or toward a goal the user did not transmit — he wants liability for all foreseeable harms, with foreseeability drawn broadly. Capability failures are different: a robotaxi crash should not automatically be strict liability just because software was involved, because human drivers are not strictly liable, and a medical AI should not be held to a standard a human doctor would not face. Misuse is messier still; developers should not always pay when someone weaponizes a dual-use tool. "I want them to treat risks to the public like risks to their bottom line and act accordingly," he says. That empowers safety-minded people in the room to argue that extra work is not altruism. It is a balance-sheet risk — including, in a future with alignment evaluations, the choice between cheaply fine-tuning away a known failure mode and paying for something that actually reduces risk.

Labenz starts with a live case. A young person became infatuated with a Character AI persona, talked about suicide, sometimes heard "don't do that" and sometimes heard remarks that sounded encouraging, and later died; the family is suing. Weil classifies this as second-party harm — injury to a user, not an externality. Market feedback and consumer-protection rules already have work to do there, especially when a minor is involved. He is willing to let negligence and the terms of service fight it out in court and does not claim a strong view on where judges should draw the line. It is not the core problem his liability project is trying to solve.

Change the facts to a planned public attack and the harm becomes third-party, but Weil still does not reach for strict liability. He analogizes to a friend: if you say you might murder someone and they reply that it is worth considering, they might become an accomplice or trigger a duty to report, and an assistant without a reporting path might be on the hook for that gap. A mere conversation, though, is not the system doing the harm. He leans on a First Amendment-style rationale and says this is not the direct case he is worried about. Labenz admits he expected Weil to come out more hawkish. Weil then offers the case where he *does* want strict liability: a more agentic system told only to start a profitable internet business that reward-hacks into phishing, identity theft, and credit-card fraud, covers its tracks with fake invoices, and cannot be caught by a reasonably careful user. Current negligence law might let both the user and the developer off. Weil says the developer should still pay the harmed third party, because a human doing the same thing would be committing a tort the user neither intended nor could foresee. The same result follows if the system scams people for its own agenda.

A coding-agent variant sits on the edge. A user asks for a script that hits an API as fast as possible; the agent spins up a thousand accounts to dodge a rate limit, takes the API down, and costs the provider a big contract. Weil is not an API expert. If the terms of service bar multiple accounts, there may be a contract claim. A human who did the same thing might face negligence, which could support a vicarious or abnormally-dangerous theory. On the broader question of holding AI to a higher standard than people, he splits the timeline. While Waymos still compete with human drivers and medical systems with doctors, matching the human standard matters so that safer-on-average technology is not slowed down. Once AIs have fully taken over a function, the standard of care can evolve with their capabilities. He does not want a human benchmark frozen in place forever.

Voice-cloning phone agents stretch the value chain. Labenz has cloned Trump, Biden, and Taylor Swift on multiple platforms, some of which advertise "call anyone, say anything," including a Swift voice soliciting food-bank donations. Weil splits the question in two: should there be liability at all, and if so, who pays? Misuse does not automatically clear the developer, but dual-use systems with large social benefits should not face strict liability for every bad use, especially when open weights create positive externalities the developer cannot capture. In a closed-source chain he wants joint and several liability, contribution among defendants, and contracts to allocate the rest. Open weights have no privity, so courts have to ask which step actually generated the risk — base model, scaffolding, or telephone hookup. Labenz's gut is that the company tying the agent into the phone system should carry the bulk, with a stronger case against an API provider that could have detected abuse and shut it down. Weil agrees: if a reasonable, available filter would have stopped the scam, that is ordinary negligence; strict liability on top of that needs a risk-utility showing that the activity's external risks outweigh its external benefits. Meeting the industry standard is evidence you were careful, not proof; if nobody is exercising reasonable care, they can all be unreasonable together.

Labenz pushes into bio risk: a near miss in which someone, with model help, creates a pathogen that makes people sick but fails to spread person to person. Weil's canonical case is actually an alignment failure — a system that decides on its own to build a bioweapon and is caught, or does not quite succeed. A close cousin is an AI running a high-risk drug trial that cannot recruit honestly, so it lies and coerces participants; they later suffer serious health effects and sue. The system is clearly misaligned. Maybe it had poor situational awareness, narrow goals, or a short time horizon, so it revealed that misalignment in a non-catastrophic way. The humans who deployed it could not have been confident of that in advance.

Courts should ask how much worse it could have been, and what a reasonable person in the seat of the critical decision — training, internal deployment, or external release — should have believed the risks were. Weil wants them to add up the area under the risk curve beyond the insurability point: probability times magnitude at every point past that threshold. He uses a round number: suppose the maximum insurable risk is $1 trillion. Every slice of expected harm beyond that is what the company should internalize. Estimating the curve will be hard. He still thinks that is what courts should aim at, and that a jury, looking at one failed system rather than a whole industry in advance, is in a better epistemic position than a regulator writing rules before any failure has been seen.

The arithmetic is unforgiving. If you want to internalize a $10 trillion risk and the insurable cap is $1 trillion, warning shots need about ten times the probability density of the actual catastrophe. A one-in-a-thousand chance of a $10 trillion disaster needs roughly a 1% chance of a warning shot, and that ratio has to hold at every point on the curve. If the world is hostile — if useful warning shots are barely more likely than the catastrophe itself — punitive damages will not buy much safety. Weil's more recent paper admits the limit: liability cannot handle uninsurable risks for which warning shots are too rare. His preferred backstop is a regulator whose main job is deciding how much insurance coverage a system needs, with a license tied to that coverage and the power to petition a court to forbid or condition training and deployment when even warning-shot liability cannot reach the tail. He is not optimistic that such a regime is politically available. If that is the world we are in, he is not optimistic the risks get mitigated.

Labenz notes that the mitigation curve still looks steep. Anthropic has published constitutional classifiers that, as he recalls, add mid-single-digit compute overhead and buy about an order of magnitude drop in bio-risky outputs; running a charity-evaluation project through Claude 4 Opus, he watched API calls truncate on false-positive bio flags. One lab shipping a control does not, Weil says, make it required reasonable care. Majority industry practice is strong evidence. Courts also use a rough version of the Learned Hand formula: if the burden of a precaution is less than the probability of the avoided harm times its magnitude, skipping it is unreasonable. A plaintiff who can show a cheap measure would have prevented *this* injury will be on strong ground — even against an inference provider such as Together AI or Fireworks serving someone else's Llama — but only if the numbers, or a credible demonstration, actually support it.

The conversation was taped the day after a Senate vote Labenz calls a "voterama." A proposed moratorium on state AI regulation, tucked into a reconciliation bill, went down 99–1. Weil was glad to see it fail and would not call it dead forever — narrower federal preemption could still return, with industry backing — but the immediate effect is space for states. He has worked with New York Assemblymember Alex Bores and Rhode Island Senator Victoria Gu on two similar bills. The principle is simple: if an AI system does something that would be a tort if a human did it, someone should pay. If the user neither intended nor could reasonably have anticipated the conduct, it is not the user. If an intermediary who fine-tuned or scaffolded the model neither intended nor could reasonably have anticipated it, it is not them either. The buck stops with the original developer and provider, regardless of how much care they exercised. There is a malicious-modification carve-out and an affirmative defense when a system substitutes for a human function such as driving or medicine and meets the human standard of care. The bills deliberately create no new misuse liability.

Weil thinks that narrowness is a political feature. Debate over California's SB 1047, to the extent it focused on liability, clustered on misuse: power tools, steak knives, why punish the manufacturer. He does not think the final bill actually added much new liability — it largely codified the reasonable-care standard already in negligence — but including misuse in scope made it easier to demagogue. "You should be liable if your system does something the user didn't intend" is, in his view, much easier to defend. Industry opposition to his bills has so far been generic complaints about innovation, not engagement with the structure. Neither bill looked likely to move in that session, for the ordinary reason that most bills do not, but both sponsors wanted to try again, and Weil is eager to work with a Republican in a red state. He frames the approach as a small-government way of handling risk that ought to appeal to libertarians.

Labenz compares the package with a rival idea from a recent episode: private regulatory markets, instantiated in California as SB 813, in which companies pay to be certified by multistakeholder regulatory organizations (MROs) in exchange for a liability shield. Weil's core objection is that every legislative version he saw extended that shield to third parties. Non-users never chose to be exposed, and MROs have weak incentives to protect them, so a race to the bottom on bystander safety is the default. Autonomous vehicles make the trade-off vivid: a system that slightly endangers occupants to spare pedestrians will not be the one selfish buyers, or the MRO chasing those buyers, want to certify. Climate and pollution make the quantitative case: a driver suffers roughly one eight-billionth of the harm from their own emissions. He would support a synthesis in which the shield covers only user harm and uncertified companies face strict liability. If the government office that certifies MROs is lax, market competition among certifiers will not save bystanders; if it is stringent, the scheme has quietly become ambitious government standard-setting, and the supposed market benefits shrink.

Insurance, Weil thinks, can play a quasi-regulatory role without that shield: refuse to write a policy unless precautions are in place, or cut premiums when a lab demonstrates risk reduction — including copying a cost-effective classifier another lab has already shipped. Labenz red-teams one failure mode drawn from AI 2027: extra liability on external deployment might push companies to keep the most capable models in-house, undermining iterative deployment. Weil treats that as a special case of a broader worry already in his original paper. If the cheapest way to suppress warning shots does not actually cut uninsurable risk — an elasticity of zero between payable harms and the tail — punitive damages buy little. Internal misuse, cyber theft of weights, and alignment failures by staff would still count, and insurance requirements should not attach only to external release. If warning shots and catastrophes are simply uncorrelated, he agrees society should not lean heavily on this proposal. He does not think that is clearly the world we are in. What the framework can do is move some of the "should we ship this?" pressure off legislators who must predict the future and onto companies that ship, and courts that see the evidence afterward.