If I Were Governor, I Would Sign: California's Last Debate Over SB 1047

In late August 2024, two letters were circulating around the California State Capitol. OpenAI came out in clear opposition to SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. Anthropic did not formally endorse the bill, but said it was workable, that the benefits probably outweigh the costs, that the risks are real, and that compliance is feasible. Former House Speaker Nancy Pelosi also raised concerns. The bill had cleared every committee. The Assembly still had to vote, the Senate had to reconcile, and all of that had to happen before August 31, after which the bill would land on Governor Gavin Newsom's desk.

This is a Cognitive Revolution special. Host Nathan Labenz brings back the same three guests who debated the bill in May, a conversation that had been well received because people with opposing views managed to argue in good faith. Nathan Calvin represents the Center for AI Safety Action Fund, one of the bill's co-sponsors alongside Economic Security California Action and Encode Justice. Dean W. Ball, a researcher at the Mercatus Center, published a critique the day after the bill text dropped and has remained a prominent opponent. Steve Newman is building a project whose working motto, modeled on Google's old mission statement, is to take the AI world's disagreements and make them universally constructive and accessible.

Labenz states his position plainly: if he were the governor, he would sign. That was not a stance he had held through every revision. He never bought the claim that the bill would be a crazy chill on AI research, but there was a point when a rewrite seemed to grant too much power to the Frontier Model Board, and he was inclined to withdraw even tentative support. A misguided board, he thought, could easily have done more harm than good. The board is now gone. Friend of the show Zvi Mowshowitz's verdict is that the bill has primarily become a transparency bill: frontier companies already promised safety testing; the statute would force them to write the plans down, hand them over, and face an audit.

Calvin walks through the rewrite. The new Frontier Model Division is deleted; some functions move into the Government Operations Agency, or GovOps. Safety submissions are no longer under penalty of perjury, and some of them become public with room for company redactions. Non-discrimination and pricing-transparency rules for cloud providers are gone, as is CalCompute, the public-compute rider. The signature duty shifts from providing "reasonable assurance" that a model will not contribute to large-scale harm to taking "reasonable care" to prevent unreasonable risk, the background standard of product liability. Coverage is now a hard dollar line: more than $100 million to train from scratch, or more than $10 million to fine-tune someone else's covered model. Starting in 2026, covered firms must hire a third-party auditor, with GovOps writing the audit rules. Employees who find risks inside the company get a formal channel to report them. Labenz's remaining objection is that $10 million fine-tuning threshold. On OpenAI's platform a fine-tune often costs $5 or $100; almost none will hit $10 million, which leaves liability on open-weight releasers such as Meta and lets everyday application developers, especially teams selling autonomous agents that take real-world actions, off too easy. He has little patience for OpenAI's opposition letter after a year of confusion, including the strawberry-emoji hype around o1. Anthropic's letter he actually reads. He thanks Senator Scott Wiener of San Francisco for what looks to him like a herculean labor of public service.

Ball accepts the factual recap and then concedes more than opponents usually do. The bill, he says, is very much improved from February, improved enough that other California AI bills look worse. Until recently, AB 2930 would have required almost every business that used AI to write an algorithmic impact study. An electrician who wanted a model to draft bids for customers would have had to run an assessment first. That provision was walked back. Ball still has nits to pick with SB 1047, and he stacks them from the institutional weeds up to political structure.

The Frontier Model Division is gone, but leftover powers sit inside GovOps, the agency that supplies California's back-office functions: HR, IT, the analog of the federal General Services Administration. Ball realized this late, in a Twitter exchange with Jen Pahlka. That is not an entity with the institutional culture to regulate frontier technology. Its "guidelines" sound harmless because they are nonbinding, but they are soft law, informal rules that can be flexible and also arbitrary. He points to FTC Chair Lina Khan, who posted that she likes open-source AI, then added that if a prominent provider went from open to closed, the Commission would look down on that. Imagine Llama 5 looking dangerous and Meta of two minds about releasing it: a threat against closing the model, coming from a different part of government, is exactly the kind of mixed signal soft law produces.

Auditor rules could be twisted the same way. Suppose a "good audit" requires looking at usage data. An open-weight provider cannot see what downstream users do, cannot obtain the audit, and therefore cannot satisfy the bill. Zooming out, Ball's larger objection is that California should not be doing this at all. His reading of the last five to ten years of Western tech regulation is that policymakers in two capitals, Brussels and Sacramento, noticed they could write rules for American companies, and maybe the world, while the federal government sat it out. SB 1047 covers any company that does business in California, which in practice means a model distributed in the state, which in practice means the country. He does not think Texas will want to be regulated by Sacramento. People talk relatively seriously about national divorce, he says; a split over AI rules is one actual avenue. California is also in no position to handle the national-security questions the bill implicates.

The deepest issue is that this is model-based legislation. You can regulate people's conduct with AI, set sectoral standards for healthcare or banking, or regulate the models themselves. Ball thinks all three will be needed, but model-based rules are a costly break from how software has been governed: permissionless innovation, and a world in which it is surprisingly hard to sue a developer for a bug. SB 1047 is not aggressively attacking those foundations, he says, but it is lightly attacking them, and "being a little bit pregnant" is a binary. Once you step over, you are in a different policy regime, even if the first step is small. Justifying that for all frontier models requires evidence of a dramatic jump in capabilities over five to ten years. Ball has not seen it. He cites Martin Casado's argument that current progress looks like curve fitting toward an asymptote, while the real world is heavy-tailed. Self-driving cars, the closest thing so far to an AGI-shaped problem, have been stuck on that long tail for years. He sketches two GovOps failure modes. In one, hospitals, lawyers, teachers, and government employees, the most important interest group in California, use an expanding regime to pin AI down in a thousand specific settings. In the other, GovOps publishes a 36-page memo full of flowcharts, frontier labs with $10 billion clusters treat it as a box-checking exercise, everyone goes to sleep, and an unknown that nobody modeled is what actually blows up.

Newman has been watching the public conversation coarsen, and he is trying to do something about it. AI policy sprawls across capability timelines, biotechnology, the job market, cybersecurity, and a thousand other questions, moving too fast for even full-time watchers to keep up, and the fights on Twitter and Substack fragment into misunderstanding. His project would host constructive discussion and then distill it. A long thread on whether a model could help someone make a supervirus, and whether DNA synthesizers already screen that out, is not something 99.9 percent of people will follow. The site would publish a one-page explainer, and if you disagree, you come join the conversation, with a commitment that what happens in the forum shows up in the explainer.

He also names the fault line that keeps surfacing. A lot of intuitions about whether these regulations are needed, and whether they belong at the model layer, seem to collapse into one judgment: how powerful is AI going to get, and how soon? People who think it will get very powerful tend to want to do something and are more willing to pay costs. People who do not think that will happen do not feel the need to pay those costs. Labenz is not sure that accounts for everything. Anthropic and OpenAI both seem to believe very powerful AI is coming soon, and they still landed on opposite sides of this bill. Trust is the other variable. Do you trust Sacramento to keep a light touch, or do you see a regulatory ratchet? Do you trust labs' voluntary scaling policies once commercial pressure rises?

Ball treats the OpenAI-Anthropic split as jurisdictional more than cynical. OpenAI's letter spends a lot of time arguing that California is not well suited to do this. For Ball himself, the fundamental question is whether the world is looking at massive qualitative leaps every 18 to 24 months for at least the next six years. Is Leopold Aschenbrenner right? Are we going to be building a trillion-dollar compute cluster with 50 nuclear reactors in three and a half or four years? If that is true, he says, we have some work to do. He also worries about political economy either way. GovOps is not going to be the FDA of AI, but it could still be captured by threatened incumbents, or it could issue generic memos that labs with real clusters treat as theater. The revisions, he grants, have lowered his capture fears relative to the first few months of the fight.

Calvin starts from the uncertainty everyone already admits. There is real disagreement, inside science and inside companies, about whether catastrophic AI risk is real and whether it arrives on the order of years rather than decades. There is much less disagreement, he says, among the people building the systems. When he talks to people at the labs, and to Geoffrey Hinton and Yoshua Bengio, they do not think they are making another email client or social network. They think they are doing something incredibly significant that could get out of control. If the people creating the most advanced models talk that way, it is worth taking seriously. Companies have signed voluntary commitments on testing, mitigations, and cybersecurity, and departing employees keep producing evidence that when commercial pressure is intense and the duty is purely voluntary, those commitments shrink.

A statute that restates reasonable care puts the duty on the desk of the C-suite instead of leaving it as a conceptual feature of common law. Required publication of safety protocols makes bluffing costly. Whistleblower channels give insiders somewhere to go. On federalism, Calvin spent a year in the U.S. Senate and likes the institution; responding at the speed of exponential technology is not among its virtues. Handing the problem to adaptive agencies is harder after *Loper Bright* overruled Chevron deference. A court struck down the FTC's noncompete ban as exceeding statutory authority. Congress can always preempt a state bill. Waiting for it to get its act together, Calvin says, does not seem like a credible plan. He also notes a virtue of the California process: people like Charles Foster red-teamed the text in public, and some of those online critiques, including the worry that a flat compute threshold would swallow more developers as compute got cheaper, became amendments.

He is still frustrated by the May discourse that treated the bill as an OpenAI regulatory-capture play. After OpenAI came out in opposition, he did not see those people say they had been wrong. He also resists the idea that any model-layer rule requires a specific AGI timeline. An international pause or a licensing regime that demands permission before a training run carries huge costs. Asking companies to be more transparent, asking government to issue guidelines, and giving whistleblowers a channel look robust across many worlds. If the severe risks never show up, liability never bites. That, for Calvin, is the advantage of liability over licensing: it is more flexible, and it leaves testing design with the labs rather than sending government testers in. Andreessen Horowitz has been more absolute than Ball, calling for zero liability at the model layer and submitting a letter that existing cyber and terrorism laws already cover the bad user. Calvin is sympathetic to that scheme for social media and email. He is not convinced advanced AI is a technology for which putting all responsibility on the user still makes sense.

Ball's most concrete warning is a prosecutor. After a serious cyberattack, investigators often will not have the analog of chat transcripts. They might know GPT-4 was used and not know the prompts. An ambitious California prosecutor looking for a national splash brings a novel case on incomplete facts, and one judge writes the standard for whether a model "materially enabled" the harm. That opinion becomes a national precedent. Ball calls it a bomb sitting there. He also doubts the timing argument at a fundamental level: if models do terrifying things for three years, he does not believe Congress will sit on its hands to protect American innovation. Facts will change the conversation. He reads the last two years of AI policy as the safety community dominating first, then failing to tell a good implementation story once ideas hit daylight, after which the accelerationist side won buy-in on open source. Accelerationists, he adds, do not have good answers either if AGI is actually coming soon.

Ball is an accelerationist in the plain sense that he loves technology and wants more of it. He still thinks his own camp is empty-handed on AGI. Open-source the systems and let them fight each other? Get a new vaccine every day? That cannot be the plan. If we are lucky, he says, the best outcome from that world is tyranny: a strongman, or a strong AI, imposing order. He calls himself a classical liberal, and he does not think a classical-liberal story of AGI governance exists yet. Policymakers have already decided the safety community was an unreliable narrator on some points. They may decide the same about accelerationists. If AGI is real, the design problem is on the order of writing a constitution, a generational intellectual project. SB 1047, even a simplified federal version, might buy a few years. It is not the long-term strategy.

Calvin takes the other side of the same fear. Without a modest, public process like SB 1047, the default after a real scare is an antibody response from the national-security state: a Patriot Act, a small circle of people making the decisions behind closed doors instead of a year-long argument that random good-faith people on Twitter can still influence. He counts himself among people who believe in open innovation. Public polling, in his reading, is not super excited about AI, and neither are a lot of legislators. If even a relatively modest, fully public bill dies in a storm of reaction, the next intervention may be far more opaque. He is reading a book on U.S.-Iran relations and thinks of the Shah: ignore discontent long enough and you get a revolution nobody is happy with. People who will celebrate if this bill dies, he says, should grapple with that seriously. On this point the two of them almost agree: AI policy is a deeply underdeveloped field, and both sides need a better story.

Calvin's analogy is a new virus in some part of the world. Maybe it fizzles; maybe it climbs the exponential. Shutting down all of society would likely cost more than it is worth. Starting work on a vaccine, in case it takes off, looks cheap. Whether SB 1047 is that vaccine is debatable. Looking for policies that are not a bad bet in multiple worlds, he thinks, should be a common mission. He was glad to see effective-altruist-adjacent people praise METR and the AI Safety Institute's open-source safety evaluations, and even to hear Martin Casado, with whom he agrees on almost nothing, take synthetic-biology pandemic risk seriously. That is the defensive-accelerationist style Vitalik Buterin talks about: prepare for the bad tail without pretending you can pause the future.

Newman has less faith than he used to that a roaring AGI tsunami would produce constructive action. What shook him was watching the United States, after the first Covid response, fail to prioritize nasal vaccines and better monitoring for the next pandemic, work that would cost infinitely less than coping with Covid, because of policy fatigue. He worries about a world that passed SB 1047 or some federal analog, absorbed some moderate AI controversy, and then froze when the real thing arrived, locked onto the wrong path. Ball took the opposite lesson from Covid: America nationalized large parts of the economy in a week, kept empty planes flying between cities because the government said so, and revealed an underrated ability to turn on a dime, for better and worse. That craziness, he says, is why this is the only country that would even entertain open-source AGI, and why AGI itself may be a uniquely American invention. Calvin's same-day news flash is that Robert F. Kennedy Jr. had endorsed Donald Trump, with Nicole Shanahan reportedly making a public apology for Operation Warp Speed a condition. Government under pressure can do a lot, including Warp Speed. Whether it does smart, measured things that look one or two years ahead is much less clear.

Labenz wants forks that would actually make history look different if the bill passed or died. First, Meta. Under the $10 million fine-tune line, Meta would remain legally responsible for the vast majority of Llama 4 fine-tunes, and for the combinatorial explosion of apps those models get dropped into. Ball's guess is that Meta would try to geo-fence open-weight Llama the way it already plans not to release in the European Union, perhaps by a license term that says the weights are not for people in California. Calvin corrects a crucial detail: the bill covers companies that do the relevant business in California *or* are headquartered there. Meta's headquarters is in Menlo Park. Dodging the statute the EU way would mean relocating the company, which Calvin would be pretty surprised to see. Ball immediately grants the point: they are not going to move headquarters because of SB 1047. The duty, Calvin adds, is not strict liability for everything a downstream Llama does. It is an argument about whether Meta took reasonable care in light of what a reasonable person would think about those risks.

Second, Elon Musk and Grok. Labenz has a hard time seeing xAI refuse to serve California over burdens this low. Ball goes further: he speculates that Musk wants the bill to pass. Musk has said nothing about it despite being extremely online, despite caring about AI, and despite the conversation unfolding on his own platform. Dan Hendrycks, a central figure behind the bill and founder of the Center for AI Safety, is a safety adviser to xAI, and Musk has spent years warning about existential and catastrophic AI risk. Ball adds a sharper guess: Musk might be delighted to have a reason not to open-source Grok. Calvin finds a California exit similarly surprising given the actual compliance load and Musk's stated attitude toward taking these risks seriously.

Third, a technical silver bullet. Hendrycks has a recent paper on tamper-resistant model training: not a wrapper of guardrails, but refusal trained into the core weights so that downstream users cannot cheaply recover the dangerous abilities. There is related work out of China. Labenz wants to do a technical episode with Hendrycks. Newman says that if that line of research really bore fruit, it would be the poster child for the best thing one could hope for from SB 1047. Ball likes the direction and still notes the paper seemed to make models a little dumber. If you could pull it off, he says, it would be close to a silver bullet, and it would make the reasonable-care standard much clearer: an open-weight model that is extremely expensive to jailbreak shifts responsibility onto the fine-tuner who meant to undo the refusals. Calvin's broader hope is a race to the top on safeguards, turning the supposed binary between openness and safety into less of a false choice.

The tort-law underlay keeps surfacing. Ball says Senator Wiener has described SB 1047 as having profoundly narrower liability than existing law, and Ball thinks that overstates how the common law actually treats software. Licenses are full of waivers; negligence cases against developers are, in practice, hard to bring. Without the bill, AI might be read like software. The bill makes liability a certainty. Calvin agrees the case law is thin, but he thinks non-lawyers overrate how bulletproof those waivers are. Even a perfect contract binds the developer and the user, not a third party. The bank that gets hacked never signed it. When the harm is large enough, courts find someone. He points to Purdue Pharma and the opioid litigation: before those suits, massive manufacturer liability for misuse looked novel; after a catastrophe, courts have a strong instinct to locate responsibility. Ball's simplification is that cyberattacks are a negative externality of software, developers do not pay that price, and that fact is related to why software has been so successful. Academics who want software liability for cybersecurity, he says, have some of the worst ideas he has ever heard. AI might be different. He still wants people to understand that existing law has been fairly protective, and that this is a choice.

Calvin's closing is a plea not to treat this fight as a bubble. The software industry and its venture backers will not get to live apart from government and the general public. SB 1047 will not save anyone from rogue AGI, and it will not cause the harms its loudest critics fear, but it does take a first step on three questions that are not going away: whether model-layer duties are appropriate, whether states have a role, and whether AI should be treated as continuous with software policy or as something distinct. He wants people who oppose the bill to put more concrete alternatives on the table, especially if they believe the stakes are about to get considerably higher. The process, for all the online vitriol, also showed that a long, public argument can improve a text. He hopes that continues.

Ball was the first public critic the day after the text dropped, expecting a familiar state-and-local policy fight. It was not that. It was healthier, he thinks, and also the opening of a conversation that, if AGI is real, will be as complicated as writing the Constitution. On the substance he is unmoved: competence, sovereignty, and model-based liability are still the core. He adds a vibes argument he knows is not substantive. Time feels faster; both political parties are now vibes-based; SB 1047 is bad vibes in the American tech industry, and a world in which it passes and the country just starts feeling bad about itself is not nothing. More seriously, the first prosecutor and the first case are a price he is not yet convinced is worth paying. His prediction is crisp: the legislature will pass the bill, and the governor will veto it.

Newman brings it home. AI policy is hard. There are no good answers, only degrees of bad ones. That hardness is not an excuse to pass whatever is in front of you, and it is not an excuse to do nothing, because nothing is one of the bad policies. Uncertainty stacks at every level, from the technology to how a prosecutor will behave to conflicting goals and technical limits. The only way through, in his view, is the kind of constructive argument this pair of podcasts tried to host: not just "I hate this bill," but "here is how it could be better," or "here is the other thing we should do instead." Long may it continue, he says. Calvin declined to put a number on passage, citing an interest in staying out of a betting frame. The recording ends before Sacramento's final act. What the conversation leaves behind is a reusable test: read the amendments before the takes, ask whether the speaker expects AGI soon, remember the $100 million and $10 million lines, and put more weight on work that is not a bad bet in more than one future, from tamper-resistant training to independent evals to safety protocols that actually get published.