100 deaths or $1 billion: a 15-page New York bill would set a floor for frontier labs

Poll after poll, Nathan Labenz notes at the top of *The Cognitive Revolution*, shows the same pattern: significant majorities of voters in both American parties, and internationally, want more regulation of AI. Far more people worry the government will not go far enough than that it will go too far. Yet in 2025 the United States still had no meaningful federal law covering foundation models or frontier development. Part of that is priority: people agree, but it is not their top issue. Part of it is a healthy fear among legislators that they do not understand a fast-moving technology well enough to regulate it. And part of it is the fear that anything which slows American labs could help Chinese developers win an AI race.

Labenz has spent most of his adult life arguing against premature or heavy-handed rules that might deny the benefits of breakthroughs. Even so, he says, the pace of capability gains — and the fact that Turing Award winners’ forecasts still run from utopia to human extinction — ought to produce at least a minimum standard. His guest is New York Assembly member Alex Bores, who represents District 73 on Manhattan’s East Side and is the sponsor of the RAISE Act. Bores has a master’s in computer science with a machine-learning specialization and spent nearly a decade in tech, including almost five years at Palantir. Labenz’s framing is blunt: he is perhaps the tech-savviest and friendliest legislator the industry could hope for, and the bill is approximately the least burdensome regulation anyone could realistically expect to see passed.

The RAISE Act would hit only a high-single-digit, maybe low-double-digit, number of well-resourced companies. It would require a safety plan, third-party audits that the plan is followed, disclosure of strictly defined critical incidents, and whistleblower protections — in line with voluntary commitments frontier developers already made in the Biden years. Labs would still have wide latitude to design those plans, so long as they take reasonable care to minimize risks of 100 or more deaths or more than $1 billion in damage via chemical, biological, radiological, or nuclear mechanisms, or what Labenz calls automated crime. Labenz spends the rest of the conversation red-teaming the definitions. Bores defends them, and also explains how he has already compromised with industry to avoid a tangle of distinct state rules.

Bores was the first Democrat in New York state government with a computer-science degree; he is happy to report he is no longer the only one, though the club is still small. He worked in tech for nearly a decade before running, joining Palantir as a data scientist and rising to lead a large portion of the government business, then a couple of startups, while completing that machine-learning master’s. When the District 73 seat opened in 2022, friends pushed a simple thesis: he had always been downstream of policy, trying to patch it with tech; here was a chance to go upstream and design it. Terms are two years. He won a contested primary, is still friends with the people he ran against, and is starting his third year in the legislature. So far, he says, it is a place where you can actually be effective.

His tech file is a mix of copying homework and picking fights nobody expected. He pushed the state toward cloud computing so services move faster. He helped kill the contract language that let companies claim any IP a worker developed while employed, even off the clock and unrelated to the job — by copying California’s decade-old rule word for word, which was enough for the industry to support a limit on itself rather than face a new, unique compliance burden. He is working on a legal shield so companies that red-team their own models for child sexual abuse material do not get sued for the very tests meant to stop it, and he has a bill encouraging C2PA, the industry’s own open provenance standard for images and audio. The C2PA pushback still baffles him: an open standard the industry built, and then a chorus of “don’t encourage the thing we developed.”

Outside AI he is a housing and courts legislator. A land-value-tax pilot aimed at vacant New York City lots is, on his math, an $800 million-a-year swing if empty land were taxed at market value instead of the city’s discounted assessment. He passed a bill creating 20 new judges to chip at trial backlogs, then discovered he could add none in Manhattan, the Bronx, or the Albany capital region because of a cap in the state constitution that dates to 1846; a constitutional amendment to lift it is next. Most of this work, he insists, is not a culture war. The Assembly and Senate send about 800 bills a year to the governor; she signs around 600; the public hears about 20 to 50. Belief in government is low in part because the news covers fights, not water infrastructure.

District 73 runs roughly 34th to 93rd Street on the East Side, Second or Third Avenue to Fifth, plus Sutton Place: highly educated, the wealthiest district in the state, and still a place where 40 percent of renters are rent-burdened. If you polled them, Bores says, AI would not rank near the top. Cost of living, public safety, and schools would. Mention his tech background, though, and the reply is almost always the same: thank you, because I am terrified, I do not know what we should do, and I am glad someone is thinking about it. He calls it latent unease more than fear — a feeling that something is coming, without a script for what to do about it. He sorts AI politics on three binaries: pessimistic or optimistic, short-term or long-term, use-case-specific or model-general. Most of his colleagues’ bills sit in short-term pessimistic, aimed at discrimination and privacy harms already here. RAISE is the long-term, model-level bill, not because it is the only thing that matters, but because fewer people are writing it.

Labenz expected, two years ago, a unified medical front that would lock AI doctors in a narrow box. It has been slower than he guessed. He can still put a child’s eyelid symptom into ChatGPT, Claude, or Gemini — the user-side trick is to say you are preparing for a conversation with your doctor — and in his case it probably displaced a visit. Bores’s explanation is a little counterintuitive: licensed professions already have a moat. Elected officials, he notes with a kernel of truth under the joke, are among the least qualified people to obsess over AI displacement, because by law you cannot elect an AI. Doctors and nurses have the same kind of built-in protection. The jobs that turn over first are the ones with neither a license nor a union. Last year’s strikes by SAG-AFTRA, the Writers Guild, and the Directors Guild were, in large part, about AI’s role in making movies and TV.

New York’s answer for its own workforce was a law saying the government may not use AI to replace an employee, while still allowing it to replace tasks. That baseline, Bores argues, is what turns Luddites into adopters: you as a person are not going to be fired, so now learn the tool. When the city automated MetroCard sales, station agents were not cut; they were sent to walk the halls and actually talk to people. He has personally sat on hold with a government agency for four hours. A chatbot that answered him first would have taken him off the queue and gotten a human faster to the person who still needed one. Jennifer Pahlka’s *Recoding America* story about “the new guy” who had been in the job eighteen years is, for him, the point: there is a huge stock of tacit knowledge, and you do not unlock it by making the fight “us versus the machines.” Attrition and repurposing can shrink headcount later. Starting with a war does not.

How long that bargain lasts is the harder question. Self-driving is starting to work; millions of people drive for a living; “learn to code” is itself now a contested piece of advice. Bores agrees the private sector will feel it first — government has vacant slots and regulatory fat to cut — but he will not hide behind the historical slogan that every revolution creates more jobs than it destroys. The speed of each wave has been shrinking. Until recently you could retrain because the next revolution would not land inside your career. Now jobs can be replaced every year or every six months, and if the AI acquires new skills faster than any human, retraining puts you back in the same spot. That, he says, is a question government does not yet have an answer for. UBI is part of the broader conversation; a New York home-care program that pays family members is, in his telling, a way to keep people out of nursing homes and save the state money, not a stealth basic income. Once RAISE is through, a new social contract is where he wants to spend cycles.

Bores’s pitch for RAISE is short. Companies doing extremely advanced research whose impacts are not yet known should have basic safety protocols, largely the ones they already promised during the Biden administration. Four provisions: a safety plan, written down before quarterly-profit pressure hits; a third-party look at whether that plan is actually followed; disclosure of critical safety incidents, defined tightly in the bill as events that really increase risk; and a ban on retaliating against workers or that third party if they blow the whistle on truly catastrophic risk. That is the whole ask. He does not claim it is the last word. It is a floor, aimed at the moment when a public company, two months behind in an AGI race, decides to skip a test.

Who is in scope is the real design choice. A “large developer” is a company that has spent $100 million, cumulatively, training frontier models. Bores thinks that is still single digits, maybe just into double digits, and academic research is exempt because the same quarterly-profit incentive is not there. A frontier model itself is defined two ways, which he unpacks later; the headline threshold Labenz cites is 100 deaths or $1 billion in damages. The covered pathways are narrow: CBRN — chemical, biological, radiological, nuclear, with biology first by a wide margin — and crimes committed with limited human intervention, the category Labenz nicknames automated crime. Labenz has red-teamed voice agents that will clone Biden, Trump, or Taylor Swift and call any number. The models are still in the uncanny valley, he says, but automated crime is already doable at some success rate.

Two extra filters keep ordinary misuse out. The model has to be *materially helpful*. If Google and the LLM both return the same high-level Wikipedia overview of how to build a nuke, that is not liability. And a determined human twisting prompts to violate some obscure law is not the target; the test is closer to “I tell it to do a thing and it largely does it.” Behavioral harms — addiction, parasocial spiral, the smoking-like cases where people know a habit is bad and do it anyway — are explicitly out of scope. Bores uses tobacco and oil as the corporate analogy, not the user one: cigarette companies knew smoking caused cancer and denied it; oil companies knew for decades they were driving climate change and shipped anyway. If your own pre-committed tests say this is a massive risk of critical harm, the state is telling you to pause.

Before leaving the “large developer” definition, Bores adds the clause that will get picked apart. You have to have spent $100 million on training. A frontier model is then either a system at 10^26 FLOPs with $100 million spent on that model, or a model produced by knowledge distillation from such a frontier system at a cost of at least $5 million. Distillation here is not LoRA fine-tuning; it is using a frontier teacher to train a smaller student with similar, broad performance. The point is DeepSeek. New models are being trained on larger ones and introducing their own risk. DeepSeek V3, which claimed about $6 million in training cost and, in this conversation’s telling, distilled from OpenAI’s o1, would not be covered, because o1 was not itself a 10^26 frontier model. A future DeepSeek-like system that *does* distill from a true frontier model, and that has any New York nexus — App Store availability, employees, business presence — would be.

Labenz walks the logic several times to be sure. The $5 million is per model; the $100 million is cumulative over all time. Miss the $100 million compute-on-training gate and the bill does not exist for you, distillation or not. Hit it, and twenty $5 million distillations, or one 10^26 run, or a mix, can all make you a large developer of frontier models. Delete the distillation clause and you only catch individual 10^26, $100 million runs, and a DeepSeek-like phenomenon falls out of the picture. Keep it, and you still leave “orders and orders of magnitude” for personal and startup budgets.

That gap is not theoretical. Labenz was a minor contributor to the emergent-misalignment work: a team fine-tuned an OpenAI model on about 6,000 examples of insecure code and got a generally “evil” model — off-domain answers, dinner with Hitler, AIs enslaving humans — for roughly $25 on the API. Low-rank fine-tuning is cheaper than full-weight training, but the orders of magnitude between a large behavioral change and the bill’s financial gates are the point. Bores does not pretend the bill catches $25 fine-tunes. He keeps repeating that the $100 million training-compute threshold is its own door. The distillation language is there so well-resourced later movers cannot take the copy-the-frontier route and skip the first movers’ rules.

Inside the safety protocol, developers must reduce CBRN risk and account for sophisticated actors — nation-states, not a single named adversary — which in today’s practice mostly means securing model weights harder than ordinary corporate security. They also have to explain why their tests actually measure what they claim, an epistemology clause Labenz finds hard because even METR-style scaffolding papers come with caveats. The bill’s operational sentence is that a large developer shall not deploy a frontier model if doing so would create an unreasonable risk of harm. What is reasonable? Bores’s short answer is the reasonable-person standard, a tool law has used for a long time. The longer answer is the standing tension in any fast-moving tech statute: people want a checklist, and they also want the rule to still make sense in six months. He takes it as a good sign that comments split roughly evenly between “too vague” and “too prescriptive.” The design, in his phrase, is to let companies grade their own homework: write the standard when you are not under economic pressure, then be checked against it. A plan that says “we do not care about safety” is, he thinks, pretty clearly unreasonable.

On open source the bill is agnostic. Evaluate risk in context, Bores says, including the context of releasing weights; do not demand a risk-based approach for everything except open source. Closed models can be monitored and gated behind know-your-customer checks. Weights, once out, cannot. Labenz calls the neutrality intellectually honest and also notes that it makes open source harder, especially for a Meta training a Llama 4 or 5 at 10^26. Bores answers with a question: do you believe there is any capability that should not be open-sourced — a detailed method for producing a powerful bioweapon? If there is any line, declare it. He also does not think the bill changes current behavior, because nothing on the market, as far as he can tell, reaches 100 deaths or $1 billion, and every public-company board should already be willing to say its products will not. Cyber risk is not in the bill; that ship, he says, has largely sailed. Labenz’s hoped-for technical out is an affirmative safety case — interpretability, or a release with virology knowledge excised — so “reasonable” is not just a 90-percent refusal rate. Dario Amodei’s timeline, in this conversation, is one or two model generations from a needle-moving contribution to a bioweapon. Highway deaths of 100 a day are not the comparison: the 100 deaths have to come from CBRN or a penal-law crime, not accidents. The epistemic mess remains. A one-in-100,000 chance of a COVID-scale 10-million-death pandemic is 100 expected deaths, and nobody can honestly offer five nines.

Labenz floats softening the bill to “publish a plan” and ease downstream reasonableness. Repeat-offense fines top out at $30 million, a number he notes some companies have treated as a cost of getting a lawsuit off their backs. Bores’s reply is the hardest sentence in the interview: people keep asking for a compromise version; this *is* the compromise version. He says he started from last year’s public debate, accepted about 95 percent of the critiques, sent drafts to five or six major labs from roughly July and August onward, asked for red lines, revised, sent another round in December, and only published in March. Third-party audits were in the Biden voluntary commitments, in Newsom’s post-veto commission, and in the EU’s version; whistleblower protections are the piece every state is already moving on; critical-incident disclosure is how you keep New Yorkers safe. On auditors, Labenz has lived through financial-services capture and has had Apollo, METR, Palisade, and FAR.AI on the show: sincere people whose access exists at the pleasure of the labs. Bores does not want to license auditors in advance. He analogizes to SOC 2, notes that the covered companies already demand SOC 2 of their vendors, and says he will watch. Whistleblower fines are $10,000 per employee per retaliation plus injunctive relief, including possible reinstatement. He agrees the dollar figure is nothing against these balance sheets; $10,000 is New York’s cross-industry standard, and raising it is treated as a Pandora’s box. The real work is existing case law: your job and your responsibilities are protected. RAISE’s only additive is that catastrophic risk, not just already-illegal conduct, is a protected report.

Incident reporting is easy to over-read. A clause covering a frontier model that autonomously does something the user did not request would, taken alone, fire constantly in a world of coding agents. Bores’s reminder is that all four listed incidents count only if they increase the risk of critical harm. An employee who leaves a machine unlocked and a janitor who sees code is not a report; China stealing the code is. Bing’s Sydney launch in India, which skipped an agreed safety-board process, might qualify in a future, more powerful case; a monitored, temporary feature that is then turned off probably does not, unless you are playing with fire. The sharpest clarification is internal use. Labenz thought deployment meant shipping to customers. Bores says the bill’s definition of deploy includes using the model, not only making it available. Testing, development, evaluation, and use to comply with law or a federal project are exempt. Ordinary internal use is not. That is, in Labenz’s phrase, the gain-of-function vibe: a guardrails-light model used in-house for AI research. Bores agrees more of the risk is now internal, which is why they wrote it in. The field, he thinks, is already moving away from assuming the most dangerous models will be open-sourced.

The last standard objection is that state law is the wrong layer: 50 different rules, and then China. Bores agrees it should be federal. “Come back to me when it is.” Federal law already preempts; any member of Congress is welcome to take his bill. The objection, he says, has more weight once Congress is doing something, and once one state has passed something. No state has. About six are pushing some form of frontier-model rule, and the sponsors are on a group text. He copies homework in the other direction too: short-term consumer bills based on what Colorado already passed, because they got there first. Before you draw a line between two points, get the first point on the books.

Session ends June 17, a little more than a month from the taping. The bill has to clear multiple committees, both houses, and the governor. New York’s distinctive extra is the chapter amendment: the governor can sign with a memo that she has agreed changes with the sponsors, then those amendments are introduced at the start of the next session. The June text does not have to be the final text. If industry and academia converge on a better definition of a frontier model, Bores says he is happy to swap it in; if the field moves between June and December, they can act. California’s bill passed the legislature by a lopsided vote and died on Gavin Newsom’s desk. New York, Bores wants people to understand, has a valve California did not.

He is not interested in an Eye-of-Sauron fight, a joke he is especially careful with as a Palantir alum. He has incorporated, by his count, something like 90 percent of what industry associations sent. The labs are not the enemy; the point of the guardrails is to get medical discovery, the end of monotony, even cyber defense, without skipping the tests. Government, in his closing, is not a spectator sport. Decisions are made by people who show up, and industry is showing up. The bill is about 15 pages, denser than a tweet and much shorter than a federal statute. New York residents can use bit.ly/raiseactny to email their Assembly member and senator. Companies and organizations can file a memo of support at bit.ly/supportraiseny. Specific fixes go to bores@nyassembly.gov; the bill, he says, is not his baby. If the reasonable people who want some regulation stay quiet, they will not get an 80-percent bill. They will get nothing. Labenz’s last word is that the ask is not really all that much.