Ban Superintelligence or Build the Rules as We Go? Max Tegmark and Dean Ball Clash Over a Precautionary AI Regime

The debate centers on a Future of Life Institute statement released on October 23rd. It calls for a prohibition on developing superintelligence until there is broad scientific consensus that it can be done safely and controllably, and strong public buy-in. Max Tegmark, an MIT professor and president of the institute, argues that negating the statement would mean allowing companies to build superintelligence even if nobody agrees it can be controlled or wants it. That, he says, would be spectacular corporate welfare, given that "there are more regulations on sandwiches than superintelligence" in the US.

Tegmark points out that medicines, cars and airplanes all require demonstrating safety to independent scientists without conflicts of interest. He adds that 95% of Americans in a new poll do not want a race to superintelligence, and that most scientists working on the problem agree nobody knows how to keep something so much smarter than us under control.

Dean Ball, a senior fellow at the Foundation for American Innovation who helped write America's AI Action Plan at the White House, says the concepts of a ban and of superintelligence are simply too nebulous. To write a statute you must define superintelligence, he argues, and any definition risks banning beneficial technology, such as a system that has largely solved mathematics and accelerates science and AI research. In practice, he fears a ban would mean freezing models at roughly the level of GPT-N plus two.

Ball adds two further worries. Safety research on superintelligence cannot be done speculatively, because you must build at least large parts of it, so a ban would leave only a sanctioned group doing the work and risk creating a monopoly or a global governmental cartel. And all of it assumes the international cooperation needed to enforce a ban, which he doubts.

Tegmark replies that the law would not need to define superintelligence at all. As with restaurants or drugs, regulators would set safety standards, for example that a system must not teach terrorists to build bioweapons, and for very powerful systems that the company must demonstrate it can keep the system under control. If a company cannot make a convincing case that its product will not help overthrow the US government, then it comes back when it can.

He uses thalidomide as the model. The wrong approach would have been a rule banning medicines that cause babies to be born without limbs; the right one is requiring clinical trials with quantified side effects and benefits, reviewed by independent experts. Applied to AI, the government defines harms broadly and the burden falls on companies to quantify and persuade, which also leaves the engineering to market forces, as with nuclear reactors that must show a meltdown risk below a set threshold.

Ball agrees with much of the description but says real regimes are more complicated. Aviation regulators, he notes, do not ask firms to prove a plane will not crash; they require affirmative statements about subsystems and layers of corporate risk management. For general-purpose AI, he asks what the affirmative statements would be, and whether a regulator would be impossibly broad or, in democratic politics, would not stay narrow.

Ball draws on an essay he wrote on the political economy of AI regulation. Because AI adoption will challenge entrenched economic interests, a licensing regime could be turned into a cudgel against technological change, for instance by demanding proof that a model cannot cause job loss. Tegmark, for his part, presses Ball on whether he thinks it is reasonable to have zero safety standards for AI today.

Asked whether AI labs should be free to release a superintelligent system without meeting any standards, Ball says developers are already subject to civil and common-law liability for recklessness or gross negligence. The pushback is that liability is meaningless if the outcome is human extinction. Ball concedes the point: tail risks are the classic case where reactive liability fails, because damages can dwarf even the largest company's balance sheet, so public policy beyond lawsuits makes sense.

Tegmark uses the debate over COVID-19's origins to sharpen the point. Whatever the truth about gain-of-function research, suing the researchers would have been pointless because they could never pay, which is why governments restrict such work and require biosafety labs rated level one through four. He then asks why digital gain-of-function, such as the automated AI researchers and recursive self-improvement Sam Altman has been excited about, should have no binding rules at all.

Ball answers that bits are very hard to regulate, so rules belong at physical choke points rather than at the layer of the model. He cites biosafety lab levels and nucleic acid synthesis screening, a policy he worked on in the Trump administration, and says the urgency of such rules rises because of AI. In his view, we do not regulate computers, software or transistors as such, even though all of them have killed people, because they are the wrong unit of regulation.

Ball describes how the US usually handles technology: build it, gain experience, diffuse it, deal with demonstrated harms through the liability system, and gradually codify standards shaped by what has been learned, eventually through international bodies. He notes that chatbot makers already face harm-based lawsuits, and says the Trump administration renamed the AI Safety Institute the Center for AI Standards and Innovation to emphasize technical standards. Writing standards without experience, he warns, leads to rules that are unrealistic and burdensome.

Tegmark agrees that this is how cars and fire were handled, but says technology keeps getting more powerful, so at some point the learn-from-mistakes strategy flips from good to bad. Nuclear weapons are already beyond it, he argues: society does not let him start plutonium research at MIT on a promise to be careful. Artificial superintelligence, he says, has a worse downside still, since losing control to a new robot species would be game over in a way even a global nuclear war would not be.

Tegmark stresses that the choice is not more AI versus less. Tools such as AlphaFold and autonomous vehicles are not particularly risky and can continue under the traditional system, while the risky branch is racing to superintelligence within three years rather than doing it safely in twenty. He praises the Action Plan for never using the word superintelligence, and Ball says that omission reflects how hard it is to know people mean the same thing by it.

Ball recounts opposing California's SB 1047 in spring 2024. At the time he doubted that scaling pre-training would produce models posing serious bio and cyber risks, and said that a model with demonstrable system-two reasoning would change his mind. When OpenAI released o1 around the time the bill was vetoed, reasoning plus tool use pushed performance up in cyber, mathematics and biology, which shifted his risk calculus.

He says he can now draw a clear line from a reasoning model using tools like AlphaFold to a virus synthesized in a lab, so targeted regulation is warranted. That is why he supported the narrower SB 53 a year later, not only because the law changed but because the facts on the ground did. The lesson, he says, is that a regulation needs a clear link between emergent capabilities and a cognizable harm.

On human extinction, Ball says the mechanism is much harder to demonstrate concretely. If critics could name empirical tests, such as an eval a model must pass, he would happily support making that a standard, and he suspects labs would run it even without a law. Tegmark welcomes this and proposes tiered classes like drug schedules or AI safety levels one through four, with light requirements for a translation tool and much heavier safety cases for state-of-the-art protein-synthesis AI.

Tegmark predicts such a regime would cause a golden age of useful AI, flooding us with medical treatments and autonomous vehicles, while slowing only the race to superintelligence, for which nobody could yet make a safety case. He would rather wait twenty years than bungle it. He also argues that being a scientist has made him humble about predicting how things fail, since nobody foresaw what thalidomide would do.

Liron asks whether policy should prepare for superintelligence at all. Ball says the Action Plan had many authors and the term is too vague to build consensus around, though his own Substack discusses it. He argues that a GPT-7 that advances science, solves old math problems and beats humans at law and coding does not seem inherently dangerous, and wonders whether it would already count as superintelligent. As we get closer to AGI, he says, we need more specific abstractions than ones coined years ago, citing Dario Amodei's analogy of driving toward Chicago.

Tegmark replies that hype tends to hijack any term that catches on, and says the original meaning from I. J. Good and Alan Turing was machines that do everything better than us, including AI research and building robot factories. He cites a paper on defining superintelligence, co-written with Dan Hendrycks and Yoshua Bengio among others, which found GPT-4 about 27% of the way to AGI and GPT-5 about 57%. If standards only arrive in three to five years, he worries they could come only after AGI or beyond.

Tegmark also clarifies his nightmare scenario for listeners. Humanity has repeatedly thought too small, he says, and if the brain is a biological computer there is no law of physics preventing machines that do all tasks better. Such robots could build more robots without us, amounting to a new species, and Turing, Geoffrey Hinton and Yoshua Bengio have all warned of loss of control. The control problem is unsolved, he says, and the key is to build tools and go slowly with what we cannot yet control.

Ball accuses Tegmark of wanting a precautionary, preemptive regime in which some group must say yes before you can build, and says such regimes carry huge costs. Asked whether he would abolish the FDA, he says it needs deep reform: it was designed for an industrial-era view of disease as uniform across populations, whereas modern biology shows highly individual conditions that suit personalized treatment. A top-down regime locks in assumptions and entrenches interests, such as expensive clinical-trial operators.

Ball argues that general-purpose technologies like computers, the internet and software grew under lighter regulation and improved life, and that the burden of proof for a licensing regime for AI has not been met. He says that if the statement had instead called for investigating which empirical guarantees labs should be able to give, he would not have had such a negative reaction to it.

Tegmark responds with three points. First, they agree on something in biotech: nobody wants level-four labs making a lethal bird flu airborne just because people can sue later. Second, if true superintelligence takes over, it will be too late to regulate. Third, he says the statement was deliberately vague, like the early consensus against child pornography, which created the political will for experts to work out definitions. He hopes broad public agreement will let policy makers like Ball draft the details.

Liron asks Ball for his p(doom) if AI simply plays out without new regulation, defined as extinction-scale catastrophe or AI systems seizing control and killing large numbers of people. Ball answers that it is very low, sub-1% and about 0.01%. He says he has many other AI worries that seem more plausible, and that he would not worry about a hypothetical law that had the Supreme Court vote on whether each frontier model will take over the world.

Ball's concern is that real regimes would be staffed by stakeholders such as unions and would weigh jobs or economic impact, so a good, pro-social GPT-7 could be blocked. Tegmark agrees things could get political, and says signatories have very different motives: national security figures like Mike Mullen worry about control loss, people from Steve Bannon to Bernie Sanders worry about economic obsolescence and power concentration, and faith leaders worry about human dignity.

Tegmark then gives his own number. With three MIT students he wrote a paper analyzing recursive scalable oversight, the most popular proposal for controlling superintelligence, and found that in the most optimistic scenario it fails 92% of the time. If superintelligence can be legally launched with only lawsuits as a deterrent, he says, the chance of losing control is well over 90%. Ball replies that he doubts any major lab would release a model that seemed likely to overthrow the government, and Tegmark answers that thalidomide's makers also did not foresee the harm.

Ball makes two arguments for skepticism. First, preemptive regulators tend to overstate risks because they treat AI like an exogenous shock, a meteor, when society is an adaptive system. He says that if shown today's generative AI tools in 2020, you would have guessed that elections were wrecked and software engineers were gone, and neither is true.

Second, he says every general-purpose technology exhibits recursive improvement: iron makes better iron, computers help design computers, and oil and electricity are used to produce more oil and electricity. These loops can be nonlinear but have never run away, so citing AI's self-improvement is not by itself an argument for doom.

Tegmark agrees that technological progress has always involved such loops and says they explain exponential GDP growth, but argues that humans were always in the loop as moderators. A nuclear explosion has no human in the loop and goes fast. Machines that think far faster than us and instantly copy what others learn could produce a month of progress that took a thousand years before, and he says we have never built machines that replace our cognitive abilities entirely.

Tegmark addresses the argument he hears from lobbyists in Washington, that any binding regulation means China wins. He says there are two races. One is a race for economic, technological and military dominance, which is won by building controllable tools, and which he supports; the other is a race to release superintelligence nobody knows how to control, which he calls a suicide race.

He says the Chinese Communist Party values control and would not allow a Chinese company to build something that could overthrow it. He relates an anecdote in which Elon Musk told senior CCP figures in spring 2023 that a Chinese superintelligence would mean China is run by the superintelligence, and the reaction was long faces. Tegmark believes Beijing monitors its AI companies closely, and that US national security officials will increasingly see superintelligence as a threat, so both would end up building powerful tools while blocking uncontrollable ones.

Liron then asks Ball about his own nightmares. Ball worries most about tyranny and a rentier state, a small group controlling the tools of violence without democratic legitimacy, a protected middle class of rent-seekers and a large underclass with little agency. He thinks several regulatory regimes, licensing included, would make that outcome likelier. Losing the AI race to China, he adds, would be bad but is not his nightmare.

Tegmark closes on an optimistic note. He describes two paths: a pro-human future of ever better AI tools under human control, which cures diseases and makes people more productive, or a race to superintelligence in which humans cannot earn money and depend on handouts. He calls the second unambitious, and says the signatories, from Steve Bannon to faith leaders and Chinese researchers, share one thing: they are all human and want humans in charge.

Ball says most doom debates assume their conclusions. Superintelligence is a grammatical sentence in English, not proof that the bad version will be built, and the future is usually stranger than anyone imagines. He urges keeping the law adaptable, considers worlds where humans thrive alongside systems better at some intellectual tasks, and asks for a higher standard of evidence before saying this time is different.

Ball also notes that the founders deliberately made new laws hard to pass, being wary of raw democratic impulse, and that laws are rules backed by a monopoly on violence. He expects a new regime to have tons of side effects and to forgo wonderful possibilities. Instead of off-the-shelf regulation, he says, we should build a society and institutions capable of evolving with the technology, taking the risks seriously without becoming a radical in either direction.

In his wrap-up, host Liron Shapira says the policy gap comes down to p(doom). If the two met around 25%, he suggests, they would converge on similar ideas, such as tight security on development. Tegmark and Ball differ on whether AI is just another technology, he says, and the stakes are high while timelines are short.