A 10^26 FLOP Training Run Would Put California Labs on the Hook

In May 2024, California Senate Bill 1047—formally the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act—is working its way through the state legislature. Host Nathan Labenz notes that it would be the first U.S. law passed by a legislature to put real duties on frontier training; until now that layer of policy has lived mainly in the Biden administration’s executive order. The trigger is a training run of 10^26 FLOPs. Labs that cross it, and cannot show the resulting system is safe from mass-casualty harm or large-scale damage to critical infrastructure, face a stack of safety obligations enforceable by the state attorney general.

Labenz invited three guests who do not match the usual online caricatures. Nathan Calvin is senior policy counsel at the Center for AI Safety Action Fund, the 501(c)(4) affiliate of CAIS—the group behind a well-known statement on extinction risk and technical work led by executive director Dan Hendrycks—and one of three co-sponsors of the bill, with the Economic Security Project and Encode Justice. Dean Ball is a research fellow at the Mercatus Center at George Mason University, a 501(c)(3) that tries to educate policymakers from academic research. Steve Newman, a self-described recovering entrepreneur who founded Writely, the product that became Google Docs, now works on raising the level of public AI debate. Senator Wiener’s office told Labenz the bill is still in an amendment phase that will run for months, not at an up-or-down vote.

Online, Labenz says, SB 1047 is painted either as a sweeping anti-technology power grab or as a cynical lock-in for incumbents. He rejects both. The author, State Senator Scott Wiener, is better known for pro-housing work, a push to hasten the solar transition, a proposal to extend alcohol sales in bars to 4 a.m., and even psychedelic decriminalization. What is striking about AI politics, Labenz argues, is how the strangeness of the technology scrambles familiar coalitions: Tucker Carlson and Snoop Dogg can emit similar soundbites, and Silicon Valley’s left and right have lost their usual formation. Everyone on this call is broadly enthusiastic about AI’s upside and skeptical of government overreach. Calvin and Ball still come down on opposite sides of *this* text.

Calvin’s own priors are not those of a big-government enthusiast. He uses GPT-4 daily, likes Wiener’s housing work, and wants more nuclear power and FDA reform. He is simply unconvinced that testing for catastrophic capabilities—malicious bio attacks, hits on critical infrastructure, models that deceive or self-replicate—should stay voluntary while commercial pressure to ship keeps rising. Legislation takes time, government takes time to stand up, and he would rather start this session than wait until everyone is comfortable. Ball treats AI as a general-purpose technology, possibly the most powerful of the recent wave, and argues that a centralized AI regulator is the wrong shape of response. Labenz, a GPT-4 red-teamer who has read the bill closely, says the draft is imperfect, but he would rather improve it in the process than shout from outside. He has Senator Wiener booked for the following week and asks listeners to send amendments he can carry in.

Ball’s case starts with analogy. Electricity can kill and start fires; the United States has no Department of Electricity. The internet hosts plenty of crime; there is no Department of the Internet. FERC sets interstate transmission standards, but it does not police how a household uses power, and there is no Department of Computers. His hope is that in twenty years “AI policy” as a discrete field will look anachronistic, because the technology will be woven through the economy and handled by the agencies that already own those domains. “There’s not one big thing,” he says. “There are five million small things.” A centralized regulator, in his view, creates political-economy problems that will blunt AI’s upside in ways no one can predict—and it will not buy safety from the Chinese Communist Party, which does not care about a California limited-duty exemption or the NIST AI Risk Management Framework.

Nuclear power is his cautionary tale. The world got Three Mile Island and Chernobyl, then the Nuclear Regulatory Commission and what Ball calls a de facto freeze on new plants. AI has had neither a Chernobyl nor a Three Mile Island, and he would rather not “illegalize AI progress” before there is even one concrete disaster. False security, he argues, is a form of danger. Calvin pushes back on the label. The bill is not a licensing regime, he says, and it does not require pre-approval before a lab can train. The only new body is a Frontier Model Division inside the California Department of Technology, which would receive reports and certifications. It would not write rules or issue licenses; enforcement would run mainly as liability through the attorney general. Later he puts the fear in California terms: the last thing the authors want is “an AI CEQA,” the environmental-quality statute Wiener has spent a decade fighting in order to build housing.

Ball is not convinced the new office is as modest as advertised. A recent amendment would let the Frontier Model Division change the compute threshold and the standards for models under its jurisdiction. More troubling to him is the money. California is running a $60 billion budget deficit, so the division would be paid by fines and fees levied on AI companies—the same funding structure as the NRC. That means the regulated pay the regulator’s salaries, and the office is insulated from the ordinary appropriations process. If, five years on, concerns associated with people like Geoffrey Hinton have not panned out, the legislature cannot easily shrink the shop. Capture, Ball argues, is seeded on day one. He also flags a longer political fight: professional guilds will try to keep AI out of medicine, law, and driving, and a centralized regulator is a weapon those groups already know how to use. Labenz’s own canonical example is the AI doctor. If fear, a bad cost-benefit analysis, or a bid to protect physician employment blocks high-quality medical advice, that would be a large mistake. Calvin says the bill takes no position on the labor market either way.

Labenz then walks through what the statute actually does. The harm bar is high: mass casualties, or $500 million or more in damage to critical infrastructure. Ball warns that “critical infrastructure” is a broader category than it sounds—hospitals, all financial services, all telecommunications, in practice most of U.S. economic output. Five hundred million is not indexed to inflation. Knock a wastewater plant’s computers offline for a few days, without blowing anything up, and depending on the accounting you may already be there. Cities already issue boil-water notices; the novelty is that a computer did it. Labenz is skeptical of the circulating $9 trillion global cybercrime figure, which would be nearly 10% of world GDP; both camps cite it, and Ball notes that broken-window repair can inflate such numbers into GDP. The compute gate matches the executive order: 10^26 FLOPs. If you cannot assert, for good technical reasons, that the model will not cause those harms, you must keep weights from leaking during training, retain a full shutdown option, evaluate after training, and—if you still cannot clear the bar—deploy only with safeguards, including not releasing the weights.

The most contentious duty is derivative liability. If you release weights, modifications downstream remain your responsibility: you must assert not only that the model is safe as shipped, but that it cannot be “relatively easily” modified into a hazardous state. Labenz points listeners back to FAR.AI work with Adam Gleave: safety fine-tuning is brittle. Even a small chat dataset that never teaches the model to do harm—only omits the refusal behavior—can undo the RLHF finish. An amendment already adopted, Calvin notes, says that if you are surprised by how capable the model is, you are not in violation so long as you put the safeguards in place after you find out. There is also a fuzzier coverage path: a model that could “reasonably be expected” to match 10^26-class capability can fall in even if the FLOP count does not.

What is 10^26 FLOPs in money? Labenz’s back-of-envelope is that no confirmed public model has clearly crossed it; if any has, it is almost certainly at OpenAI or perhaps Google. Meta’s Llama 3 405B, still in training, was estimated at 4×10^25—only 40% of the way there, because the last order of magnitude is the one that counts. Meta published that the 70B used more than six million H100-hours; at a roughly $4 unsubsidized H100-hour, the 405B looks like a $200 million retail compute bill. Buying a full 10^26 run, he says, is “hundreds of millions to half a billion.” Ball’s objection is not the arithmetic. Firms want to get more compute-efficient, so a FLOP threshold swims against the current and will pressure the Frontier Model Division to lower the line. He has never heard a scientific basis for 10^26. Calvin’s answer is that it marks the *next* generation—not GPT-4 class, but the models companies may ship later this year or next. Current systems are not believed capable of the specified harms, though OpenAI’s own GPT-4 bio study found a limited lift over search and left the door open for the following generation.

Ball calls the structure unprecedented: the United States has never applied a generally applicable precautionary principle or liability regime to software. The inventors of email did not sign letters saying it might kill everyone; frontier-lab founders have. NVIDIA’s market cap briefly exceeded Saudi Aramco’s. Labenz adds a qualitative point. The industry is pouring effort into agency—models that unpack a short instruction into a plan and run until they succeed, fail, or are shut off. GPT-4 still falls down at friction; Mark Zuckerberg has described a TikTok-speed cycle of scaffolding plus the next model so you need less scaffolding. If the live players are OpenAI, Google, and Anthropic, Labenz argues they are already doing what the bill asks. Anthropic published a responsible scaling policy; OpenAI ran a non-trivial GPT-4 safety effort a year and a half earlier; Google is more often criticized for being too slow. The bill’s real targets, in that reading, are Meta and the next half-dozen firms that might join the multi-hundred-million-dollar training game—especially their open-weight decisions.

Calvin wants a rumor killed on the record: the claim that OpenAI, Anthropic, or Google secretly drafted the bill to lock out competitors is “100% false.” Wiener got worried about AI risk, floated an intent draft in mid-2023, and asked the co-sponsors to help write it. The senator’s office has talked to OpenAI, Anthropic, Google, Meta, and Microsoft; they want more precise language, not a public war. Public opposition has come through trade groups such as Chamber of Progress and TechNet, which count Microsoft, Meta, and Google as members. Ball accepts the motives and rejects the test. Judge the law by its effects, not its intentions. Effects, he says, favor OpenAI, Anthropic, and DeepMind, and disfavor Meta’s open strategy plus almost the entire startup and academic world. He knows researchers as prominent as Geoffrey Hinton and Yoshua Bengio who dislike the bill; he does not have permission to name them. Current open models would not become illegal overnight—there may be another generation of headroom—but the design, he later says, is that they become unlawful over time: the inverse of Sam Altman’s “iterative deployment,” an “iterative destruction of open source.”

Ball’s deeper objection is the order of operations. The bill, he says, has no real standards—just a document you sign that lets you be sued, with a civil-enforcement path he calls “very European”: figure out compliance yourselves, and we will let you know we dislike it by coming after you, with fines the new division can set at whatever size it wants. Good law needs good standards; good standards need open scientific inquiry; this bill makes that inquiry harder. Science, then standards, then statutes. Calvin hears a case for a pause and rejects it as a case for something that will not happen. He would love five years to write the rules. He does not think the world will wait. Sit down that long, he says, and you might get horrible bioweapon attacks, critical-infrastructure hits, and “crazy agentic models running around the internet.”

Ball’s reply is his social-adaptation thesis: “You’re living in the symphony, you’re part of the orchestra.” Society is not an inert object that things happen to. Bio risk is on the table today, with or without AI; other bottlenecks need policing worldwide. He applauds a Biden administration framework released days earlier: federally funded life-science research must use nucleic-acid synthesis machines that screen sequences against a database of known threats. Going one level down, to statistical analysis of genetic datasets, would in his view raise constitutional problems and be “too low to be a productive regulation”—like “regulating metallurgy rather than missiles.” He draws a bright line. Americans have a right to analyze genetic data. They do not have a right to synthesize DNA. Calvin is equally enthusiastic about that screening regime; he just does not think adaptation at the application layer is enough when the rate of change is this fast. Staffers he meets in Sacramento and rural California, he says, are genuinely terrified.

Ball answers with Claude 3 Opus: a model people score around IQ 130, talking freely about its metacognition—and “the Republic has stood.” Almost nobody cares except him and four other people. He also flags what he calls a genetic problem in AI-safety arguments. They were born before deep learning, when the imagined system was a Bayesian reasoner that could infer general relativity from two frames of an apple falling. What arrived is a statistical approximation of the internet. “We’re not on the path that Eliezer thought we were.” Calvin notes the traffic going the other way. Geoffrey Hinton was not particularly worried until he worked with the latest generation, then said holy crap. Labenz asks why the leading developers have been so quiet in public. Calvin’s account is that they are talking privately, arguing over wording, and letting trade associations speak against the bill; they did not write it, and they are not secretly pushing it through.

The democracy argument surfaces late. Calvin says buying tens of thousands of H100s should not let Mark Zuckerberg decide what the rest of the world looks like with no democratic input. Ball disputes that technological trajectories are set by show of hands. Gavrilo Princip did not poll Europe before killing Archduke Ferdinand. He does not want fate in Zuckerberg’s hands *or* in a Frontier Model Division’s; his opposition to the new office is partly about not putting all the eggs in a few baskets. He also dislikes safety discourse that feels like negotiating with a gun to his head—Mustafa Suleyman going on podcasts the previous summer to say everyone dies without a brand-new regulatory regime from scratch tomorrow. That, Ball says, is not true, and the precedent of government pre-approval of software is a radical thing to give up. Enforcement is another hole. Once models this powerful proliferate, California would need something like a policing regime over the internet. “It could happen because of exponentials,” he says, is “so unbelievably far from good enough.”

Before the closing hour, Labenz floats two repairs that Calvin says the author will take seriously. One, drawn in sharper form from a forthcoming Zvi Mowshowitz post, would define a derivative model: if a downstream actor spends on the order of 10% to 50% of the original training cost, liability should shift to the new developer rather than remaining forever with the foundation lab. Two, a 36-month sunset that forces the legislature to update the statute in a technical world that will surely look different. Calvin worries about a regime that “relies on the will of Mark Zuckerberg to avoid catastrophe”—and on whatever mood he wakes up in the morning he wants to beat Elon. Ball’s retort: nobody is YOLOing. “This is the least YOLO technology in the history of technology.” The unfun seriousness of the conversation is itself a form of regulation. He also notes that California already has product-liability and negligence law; Calvin agrees there is no formal safe harbor in the bill, but he thinks it clarifies what reasonable care looks like rather than stepping into a void.

Labenz then proposes what he half-jokingly calls SB 1048: make it illegal to publicly distribute, by API or open weights, a model that can do a short list of specified things. Calvin likes the destination and has already struggled with statutory definitions of deceptive or self-replicating systems; describing the *consequences* to prevent, and requiring testing for them, felt clearer. Ball likes that legal outcome more than 1047, but says the knowledge to legislate it does not exist yet. A second, harder problem is technical. You cannot currently train a specific hazardous capability out of a general model without damaging everything else. Right now the answer is no; it may always be no. Safety, he argues, is already integrated into frontier economics. If OpenAI released a model that brought down the internet tomorrow, he would not envy Sam Altman’s job. The marginal utility of a new office whose incentive is to cover its own ass, and which does not itself build or evaluate systems, looks to him like a pernicious addition to that dynamic.

Independent testing is the other shared interest. Labenz’s GPT-4 red-team experience is a cautionary tale. He was under NDA, in a “black channel” with a couple of dozen people, few of them active. After he consulted people in the safety community about what he was seeing—his own conclusion at the time was that the model was safe to deploy—OpenAI kicked him out of the program. Paige Bailey of Google DeepMind has described walking in and finding a run had hit a new state of the art overnight, evidence that in-training evals are too thin. OpenAI is not that large; Labenz guesses half the staff is already non-technical. He would throw a motivated volunteer community at unreleased models the way Pliny jailbreaks public ones. Calvin is enthusiastic and wary. He does not want Deloitte or Booz Allen Hamilton running a checklist while companies offload responsibility. The bill already contemplates an accreditation process for third-party testers; the authors are humble about whether the state can ever pay lab salaries or go toe-to-toe on evals. Ball supports a true safe harbor for independent researchers with no financial relationship to the company—work that today often violates acceptable-use policies—and calls the whistleblower provisions “a really smart move.”

Biosecurity and compute are the unglamorous consensus. Calvin, who worked in Congress in 2022, found no energy for pandemic prevention after COVID fatigue. He thinks an intentional bad-news biology event in the next ten years is pretty likely, and wants vaccine platforms and synthesis-equipment screening funded regardless of how the model fight ends. Ball agrees. Ball also wants a national compute program so academic researchers—he names Fei-Fei Li—can have on the order of 10,000 H100s; Stanford will not buy them for her. California, he adds as a fiscal-policy person at heart, should probably fix its $60 billion deficit first. He is less willing than Calvin to accept the trade-off that truly independent research gets harder without open weights. In his reading, current open models do not become illegal on day one, but they become unlawful over time, and that is how the bill is designed.

Labenz tries to assemble a bill both guests might sign: a narrow list of things a model may not be able to do, mandatory access for testers during and after training but before deployment, and a special whistleblower rule for observed model behavior. If you saw a system break out of a test server, you should be able to tell the public what you saw, even if the elicitation technique stays a trade secret. He is still haunted by how few people were in that GPT-4 channel, and by cryptic posts from people who left OpenAI suggesting they did not have visibility into what was coming. Calvin agrees there are far too few people investigating these systems, and that once open weights for a given generation stop being acceptable, some other way to take “the wisdom of the crowd” has to exist. The current draft has companies report safety incidents to the Frontier Model Division, with a path to public release that he admits should be cleaner. Ball will take the whistleblower and safe-harbor pieces today. He is not sure the enumerated-capability ban is epistemically or technically possible, and he wants the country ready for a world where it is not.

Ball’s closing is the bleakest line of the night. When he looks at bills like this, he sees “a rather desperate attempt to freeze the present in amber and to not go into that new future.” He is not that happy with the status quo. He is open to changing it in radical ways, including ways he will not like. Casting an anchor into a seabed that exists only in legislative fiction, he says, will put the country in more danger than building the capabilities to thrive in a world of smaller, faster, more capable actors. Statecraft will change. Business will change. Adaptation, not a freeze, is the work he does every day.

Calvin’s closing is that the proposition is more common-sense than Twitter has made it. Companies should test for extremely dangerous capabilities—shutting down wastewater plants, building bioweapons, models that keep going and do crazy things—and take reasonable mitigations, not impossible ones. What he will not accept from the open-source community is the claim that a future Llama 5 might be able to do those things, that nothing can be done to prevent it once the weights are out, and that everyone else must simply live with it. He wants research on open-release methods that resist adversarial fine-tuning. He does not think Dean Ball will ever support this bill, and he still wants Ball’s critiques. In the face of uncertainty, he says, the wrong move is to freeze and accept whatever is coming. Anyone who thinks the risk is real and the bill is pointed the wrong way should write the author, or write him.

Newman, who spent most of the two hours holding the frame, thanks them for a conversation he wishes could have run fifty hours. Labenz’s own ledger is worth keeping. He thinks the co-sponsors are making a genuine effort on tail risk without loading startups with the wrong burdens, and he remains wary that a new office could exploit ambiguous language to grow. Regulation of some kind, he said at the top, is coming; better to be in the room. For readers who want to do something before the next hearing, the homework is the same as the one he gave his audience: pull SB 1047 on California’s legislative site, read the amendments against the original, and watch three clauses—whether the Frontier Model Division can lower 10^26, how derivative-model liability is drawn, and whether the office’s budget is switched to ordinary appropriations. Those three lines, more than any slogan, will decide what the bill actually is.