Jail Only for Perjury: Scott Wiener Walks Through California's Frontier AI Bill

SB 1047, California State Senator Scott Wiener's bill on frontier AI, uses the same compute threshold as the Biden executive order: models trained with 10^26 FLOPs, plus future models that reach similar capability with less training compute. In 2024 that is a several-hundred-million-dollar undertaking, and the number of companies in the world that can touch it can be counted on one hand. Nathan Labenz, host of The Cognitive Revolution and an AI investor, calls the definition complicated on purpose. Nobody is sure how the technology will evolve.

The politics around the bill are awkward. Wiener represents San Francisco, the heart of American AI. TechNet, the trade association for Meta, Google, Amazon and their peers, is publicly opposed; none of the giants have signed on in support. Startups are among the names willing to be counted. Many other people, Labenz and Wiener both note, will say in private that the direction is right and then stay quiet, because fundraising makes a public stance expensive.

Labenz had already hosted a steelman debate between Nathan Calvin of the Center for AI Safety Action Fund and Dean W. Ball of the Mercatus Center. He describes himself as an adoption accelerationist and a hyperscaling pauser: he wants AI primary-care doctors — Google's Med-Gemini is, he thinks, a glimpse — and he also thinks the road to superhuman systems deserves a testing step. He came into this interview inclined to support the bill, with one change that would turn cautious support into enthusiasm: a real requirement for independent third-party testing.

Wiener did not come up through AI. He moved from the East Coast to San Francisco in 1997, a gay Jewish lawyer looking for community, spent years in practice, won a seat on the Board of Supervisors in 2010, and reached the state senate in 2016. The district covers all of San Francisco and part of San Mateo County, about a million people. He chaired the housing committee, then the budget committee — "a massive deficit, which is always a lot of fun" — and has spent years knocking down barriers to building homes. Healthcare access, climate and energy, and criminal-justice reform sit on the same list.

Two earlier bills are the relevant precedents. After the Trump administration repealed federal net-neutrality rules in 2018, he carried California's replacement, which became a national benchmark. In 2023 he passed a law requiring large corporations to disclose carbon emissions, including supply-chain emissions, with audits by firms certified by the California Air Resources Board. That disclosure-and-audit structure is the template he keeps reaching for on AI, even though model weights are a more sensitive secret than a carbon inventory.

The motive, he says, was the people around him. San Francisco is full of senior researchers and frontline engineers. About a year and a half before this conversation they started telling him the same thing: policymakers should get ahead of safety instead of chasing the horse out of the barn. He is enthusiastic about AI for climate and healthcare and does not want to kill innovation. In September 2023, before the legislature recessed, he published an outline and left it up for months of comment. The formal bill arrived in February. The claim that it was fast-tracked, he says later, is false: nine months, regular order, still open to amendments.

If you are training a model at that scale, the bill asks for a safety evaluation before training and before you make the model available, red-teaming among the allowed methods. If the evaluation shows a significant risk of catastrophic harm, you take reasonable mitigations. The goal is to reduce risk, not to erase it. You then certify to California's Department of Technology that you did the work. Catastrophic harm is written narrowly: nuclear, chemical, or biological weapons; $500 million or more in critical-infrastructure damage; or cybercrime in that same damage band. Wiener is talking about large-scale destruction, not petty harm. Whether $500 million should be $50 million, or higher, he treats as a negotiable number.

Enforcement is narrower than the online argument. A civil suit can be brought only by the California attorney general, and only if the model actually caused catastrophic harm and you either skipped the testing or the testing already showed a serious risk and you did nothing. The only criminal piece is perjury: intentionally, maliciously lying to the government, the same theory as lying on a driver's-license application. "It's not about if you [make a] good-faith mistake or honest mistake or you [file] something that was even something sloppy." Labenz restates it in lay terms: do reasonable testing, report honestly, and a later accident can still mean a lawsuit by the AG; criminal liability requires covering up what you knew.

What the bill does not cover is almost a second statute. Labenz notes professional licensing, workforce impact, and algorithmic discrimination are all absent. Wiener's answer is that one bill cannot do everything. This session the legislature is already moving bills on algorithmic discrimination (from a colleague in the Assembly), watermarking of AI-generated images, AI-generated revenge porn, and government use of AI. Yuval Noah Harari's rule that AI should identify itself as AI is, Wiener thinks, an active topic this year and may already have a bill. SB 1047 is the safety-evaluation bill for the largest models, nothing else. On jobs he is unusually blunt. In a perfect world, AI would mean less work and shared benefits. This society is bad at spreading benefits around. He does not want huge economic gains enjoyed by a few while most people are worse off because they no longer have work. That is a mega-issue. It will not be solved here. It still has to be watched.

"Light touch" is the phrase Wiener will not stop using. To keep it, he rejected a stack of harder ideas. Some people wanted a private right of action so anyone could sue a developer; he limited enforcement to the attorney general, whose scarce resources, he argues, will go after truly bad actors. Some wanted a license from the state before you could train or release. He does not want the government sitting in the middle of every deal. Some wanted certain kinds of models banned. He did not go there. Some wanted strict liability: cause a harm and you are automatically on the hook, reasonableness irrelevant. He rejected that too.

On risk itself he is almost folksy. Eliminating risk, he says, is usually not possible in life. Try to wipe it out and you wipe out a lot of the joy and innovation; "we all [would] be sitting in a coffin in our basement doing nothing." Life is about risk. The ask is that people at least try to reduce it when a model of this scale might cause catastrophic harm. Labenz later floats a simpler alternative: just make it illegal to distribute AIs that do certain things, biological weapons being the usual example. Wiener's reply is that we do not know the capabilities two, five, or ten years out, and even a ban list would be a list of things we may not yet be able to name. Require introspection first. Whether to restrict particular capabilities can stay an ongoing conversation.

The favorite objection on the internet is regulatory capture: large firms write a law that raises the floor and crushes small labs. Wiener denies it in so many words. None of the big companies are supporting the bill. He is in conversation with them for feedback; so far nobody has signed. TechNet, their trade association, is opposing it. The ideas, he says, came from technologists and people who work on AI safety, not from the labs. A number of startups are supporting it, which he thinks speaks volumes. In Silicon Valley he has also learned a quieter ecology: people who will say this is the right approach, thank you, and then refuse to say so publicly because they need to raise money.

The other common shrug is that the leading labs are already doing this — model cards, Anthropic's responsible scaling policy, OpenAI's answer, Google moving that way, even Meta's testing of open models — so why legislate? If a lab is already doing reasonable testing, Wiener says, the bill's impact on them is very limited. Not every lab is in that group. He has seen public argument, including on Twitter, about whether Meta's safety testing is where it needs to be; he stresses that he is repeating a discussion, not rendering a verdict. Any industry, in his experience, needs a minimum standard. You cannot just trust every lab to do the right thing. Some will go above and beyond. That is not a reason to skip a baseline for models this large.

His legislative habit, which he offers as a closing note as well, is to listen even to opponents. There is a school of bill-writing that says if you are fighting me I will not hear you. He does not subscribe. "Even if you are fighting me on a bill, if you then come forward and say hey there's something you missed… there are times when I'll say oh my God yeah you're totally right thank you… we changed the [bill]." The opponent may keep opposing. He still wants the text to be as good as it can be.

The open-source community is the conversation Wiener says he is in most intensively. He does not want to eliminate open source; he thinks it has enormous value for innovation and access. The team is working through concerns line by line. His working view is that applications built on a powerful open model are often small, so the original developer is still the person best placed to do the safety analysis. Labenz reads the current text as leaving derivative models — anyone who continues training downstream — on the original author's hook, and floats a cutoff: extra compute on the order of 10 percent, and responsibility moves. Wiener calls that a fair conversation and is open to exploring it.

Labenz also asks about the longer slope. Compute gets cheaper. A few years from now a 10^26-equivalent model might cost millions instead of hundreds of millions, and clubs rather than corporations could raise it. Absent a breakthrough that lets you prove safety, does that mean a powerful open model simply cannot be released? Wiener does not take the hypothetical. Right now, and for the future he can see, models of this magnitude are still terrifyingly expensive. Asking for a safety test is, he thinks, a reasonable starting point.

Independent testing is where Labenz is least willing to let go. He was on the GPT-4 red team, dropped everything else, and found himself under NDA, dependent on OpenAI's goodwill, with nowhere to take a finding. Access for AI-safety review organizations today, from his background conversations, ranges from nonexistent to extremely limited; even the UK AI Safety Institute, despite voluntary summit commitments, has not gotten advance access to frontier models. He started his career in mortgages just as that industry blew up the economy, and he watched nominal watchdogs decay into rubber stamps. Under winner-take-all pressure — switching from OpenAI to Google to Anthropic is one line of code — a lab can do "reasonable" testing that still misses what a motivated outsider would find. The bill currently says third-party testing "when appropriate." Wiener admits that is a little vague. He is open to a larger role, and he wants to know who the testers would be: a big accounting firm checking a box, or someone real. Model weights are core product, much more sensitive than carbon numbers, so the CARB-auditor model does not copy cleanly. Labenz's amendment is accreditation plus a protected channel to report to the government without losing access. That change, he says, would move him from cautious supporter to enthusiastic one.

Wiener's only real ask at the end is that people not treat Twitter as the text. The bill is public. He has watched a week of inaccurate posts, not all of them malicious — people hear something and repeat it. Reach out and ask, he says, before assuming that is what the statute does. People who want a stronger third-party-testing regime can write his office; they can also message him on Twitter at Scott_Wiener. He will not promise agreement. He will take the meeting.

For people who actually train models, two numbers are the ones to sit with. 10^26 FLOPs is the threshold. $500 million is the catastrophic-harm line. Before you train, or before you continue training on someone else's open weights, it is worth knowing how close you are, because derivative liability is not settled. Labenz closes where he started: he wants the family doctor, he thinks Med-Gemini is close, and he thinks a testing step on the way to superhuman systems is prudence rather than panic. That is the spirit he hears in the bill, even as a longtime libertarian who knows a new agency can become a nightmare. The stakes at the frontier, and the fierceness of the competition, are why he is still inclined to take the step.