A Libertarian's Case for Regulating AI: Kill Switches, Chip Choke Points and a New Form of Life

Host Nathan Labenz opens by saying he is an accelerationist on nearly every question except one, and gives streamlining immigration as an example. Crivello, who is in the middle of becoming an American citizen, agrees and notes that the administration has been relaxing visa requirements and closing loopholes. Labenz says he would sooner back a "one billion Americans" plan than a build-the-wall plan.

The conversation turns to the Cruise incident and the nationwide shutdown of its robotaxis. Labenz is frustrated that one incident could halt the whole effort, and he worries about a chicken-and-egg trap in which self-driving cars never reach perfection because regulators never let them deploy. He reports that Tesla's FSD handled an eight-hour trip with his grandmother better than many human drivers he knows.

Crivello is more fatalistic. He recalls the old law that required someone to walk in front of an automobile with a flag, and calls this kind of resistance "part of the process" that progress eventually outruns. He says he has made peace with it. Labenz adds that most of FSD's mistakes, such as a stop sign hidden by a tree, could be fixed by changing the environment, and nobody has the will to do it.

Asked where else he is an accelerationist, Crivello answers "pretty much everywhere." He is a libertarian who worked at Uber and saw regulatory capture and cartels up close. He describes the world as rotten with institutions, private, union or governmental, that do not want to grow the pie but only to take a bigger slice of it.

He says this explains why people like Marc Andreessen react to AI regulation by pattern-matching: it is happening again, they are doing it again. Crivello understands the reaction but argues this time is different, because a new form of life is being born. He expects regulation to be messy, with gaffes, entrenched interests and 90 percent of it wrong, and says it is still worth it given what is at stake.

What is at stake, he says, may be human extinction, and he insists this is not a tin-foil-hat theory because more and more experts are saying so. He notes that the argument from authority, that experts see nothing to worry about, no longer holds. He also rejects the regulatory-capture accusation, since OpenAI was founded on the safety premise from day one, which would make it an odd plan for capturing regulators.

Labenz says he is wary of analogies for AI and prefers to call it alien intelligence, so he asks whether "a new form of life" is a metaphor or a literal claim. Crivello says he means it literally. He argues that the whole history of the universe has moved toward greater self-organization of matter, and that this may even follow from the second law of thermodynamics.

He walks through the chain: subatomic particles, atoms, molecules, stars that forge heavier elements, proto-life, RNA and DNA, cells, multicellular organisms, brains, and finally humans. He finds it strange that people treat the present as normal, with quintillions of atoms organized coherently and pursuing goals. A human is as big a leap beyond an atom or a bacterium as anything could be.

From there he argues that there is no reason to think humans are the end of the chain. Nothing in physics forbids a being as large as a planet or a galaxy, and silicon-based intelligence has many advantages over carbon-based intelligence. He hopes the transition can be managed without humans being destroyed.

Labenz asks whether more capable AI is inevitable, noting that OpenAI argues a pause would only grow the compute overhang. Crivello says it is not inevitable and sees a contradiction in the effective-accelerationist position, which holds that progress cannot be stopped and also that regulation would stop it. He cites the Middle Ages and North Korea as cases where progress was halted, and calls it "quite fragile."

Labenz presses that the current trajectory looks like the steep part of an S-curve, with hardware and algorithms both improving fast. Crivello answers with the most extreme scenario: an end to Moore's Law. He says civilization would do fine with today's chips, which still carry a huge overhang, so freezing hardware progress would not derail it.

He adds that the industry has a few choke points, namely ASML, TSMC and Nvidia, any of which a regulator could grab. He says that gives us a kind of kill switch already, since a rule requiring a new chip in every GPU could stop compute worldwide. That would be disruptive, but he judges it less disruptive than a rogue ASI.

Crivello says he is pleasantly surprised by the executive order, not for its specifics but because the government reacted quickly and the measures are not insane. He had feared rules that would require a report from anyone who installed Microsoft Office in an AI product. Instead the order talks about FLOPs and orders of magnitude of training, and he calls it a step in the right direction.

He notes that FLOP-based rules will always leak, since algorithmic improvements and fine-tuning offer many loopholes. Labenz agrees that the 10 to the 26 reporting threshold seems reasonable as a start. Crivello says ASI alignment will not be solved by asking AI companies for reports, and that regulators will eventually have to discuss the kill switch.

His proposal is a chip in every H100-class GPU that lets a centralized entity shut down all of them at once. That would not stop laptops or phones, but it would buy a few years to work on safety and alignment, and he would ideally automate detection and response, as the Russians did during the Cold War. He admits an ASI might defeat the system, and says nobody he knows has researched the engineering, though he points to the GPS chip as proof that a cheap broadcast receiver is feasible.

Crivello extends the kill-switch idea to a stricter rule: nobody should be allowed to accumulate billions of dollars of H100s, any more than enriched plutonium. He says the control must sit at the lowest possible level, and the application layer is too diffuse because any kid in a garage can build a startup. The choke point today is the silicon.

Labenz asks for the strongest case for application-level regulation, and Crivello gives it as the fork argument: forks can be used to poke people, but that is not a reason to ban forks. He says the argument weakens as the artifact becomes more powerful. He points to the AR-15, and then to enriched uranium, which no one would sell freely on the condition that buyers not harm anyone.

He notes that the executive order currently asks only that developers fill out a form saying what they are doing. In his analogy, that is like asking uranium buyers to tell you what they plan to do. He thinks the form is not enough at the scale of risk he has in mind.

Labenz asks whether Crivello has a framework for how far scaling can go safely. Crivello says he believes GPT-4 is the most critical component of AGI and that the remaining gap is engineering outside the model, not research. He compares it to Bitcoin, whose enabling technology existed for a while before anyone put it together, and says GPT-4 carries a similar capabilities overhang.

He cites small tricks, like telling the model to take a deep breath or step back, and the cognitive architecture built around it as ways to unlock more. This is why he wants open-sourcing of frontier models to stop. If Llama 3 reaches GPT-4 level, he says, "the weights are out there" and it is too late. He puts a better-than-even chance on AGI in two to eight years and expects ASI within 30 years, and says that even a 10 percent risk of doom is enough to be very worried.

Labenz is skeptical but cites Nvidia's Eureka paper, in which GPT-4 designed reward functions that taught robot hands, outperforming human experts. Crivello says he defines AGI narrowly as an AI that can recursively self-improve, which only needs to be good enough, not better than the best human. He points to Lindy writing more of its own integrations, Nvidia using models for chip design, and his belief that OpenAI is using GPT-4 internally for AI research as signs of that loop starting.

On whether the model has a world model, Crivello thinks the debate has become silly. He describes the model as strong inside a convex hull of its training data and weak outside it, then argues that prompting tricks and cognitive architecture keep growing that hull. Labenz finds the framing compelling and cites a Microsoft self-improving software improver as another sign of life, adding that he has heard people at labs say they already have AGI in a slow takeoff.

Labenz brings up the UK AI Safety Summit and the joint statements between Chinese and Western academics, and asks whether it is naive to hope for coordination with China. Crivello says it is not. Game theory says coordination is easiest with few players in a heavily iterated game, which is the situation here, and the parties can agree on a lot despite not being close friends.

He adds that enough of the choke points are held by American companies, Nvidia among them, that the U.S. could effectively impose a kill-switch chip on GPUs sold in China, as it has done with other restrictions. He is encouraged by rising public awareness, the sensible shape of upcoming regulation, labs investing in safety and alignment, and Anthropic's technical work, though he says it is not over.

Labenz frames it as the start of chapter two of the AI story. In chapter one the field mostly speculated, and now most key players are serious about the risks, which he finds hard to improve on. He ends by asking whether it matters if GPT-4 has subjective experience, and Crivello answers with the Zen response "mu": the question is unscientific and immeasurable, so it rejects the premise.