The Man Behind Humanity's Last Exam Says Superintelligence Strategy Should Copy Nukes

Sarah Guo opens No Priors with Dan Hendrycks, director of the Center for AI Safety, advisor to xAI and Scale AI, and the researcher behind evaluations that the industry actually uses. MMLU — tens of thousands of questions across dozens of academic subjects — has been the default scoreboard for large models for years. His latest exam, Humanity's Last Exam, is meant to be the end of that road. The other new document on the table is *Superintelligence Strategy*, co-authored with former Google CEO Eric Schmidt and Scale founder Alexandr Wang, which treats CIA surveillance of foreign AI programs and Cyber Command attacks on data centers as live policy options.

Hendrycks's reason for entering the field has not changed. AI was obviously going to be a big deal if you thought it through to its conclusion; other people ignored it because it was weirder and unpleasant to sit with. It looked like the most important thing of the century, so he pointed his career at it early. Because it will be that large, he wanted a way to think about it properly, channel it productively, and handle tail risks that are systematically under-addressed. At the time, he says, people were not really doing much about it.

Guo asks what the Center's role is versus safety work inside the large labs. Hendrycks's first answer is bracing for someone whose job title is safety: there still are not that many safety efforts in the labs, and he does not think labs have an extremely large role in making this go well. They can refuse queries like "help me make a virus." They can reduce terrorism risks and some accidents. They cannot choose not to race unless they want to become irrelevant in the arena. Competing with China — and maybe, later, Russia — constrains their behavior. Tinkering with design choices or adding refusal data will not change mass disruption to digital labor. Risk management, as he uses the word, is a broader problem with a small technical component.

Guo notes that lab leaders would not say they can do nothing, and asks him to separate alignment from safety. He uses safety as a catch-all for dealing with risks, including non-technical ones such as concentration of power, and even the risk of never getting really intelligent systems. Alignment is a subset: you want U.S. AIs compatible with the U.S. public, or with you as an individual. That is not enough. China can have AIs totally aligned with China, the U.S. can have AIs totally aligned with the U.S., and you still get strategic competition. Both sides will need to integrate the systems into their militaries, probably quickly, and competition will force a high risk tolerance. Reliable obedience does not make the overall situation fine. The analogy he is circling is two firms with perfectly loyal employees who still fight a price war: loyalty of the pieces does not change the game.

On national security, Hendrycks refuses to oversell the present. AI is not that powerful in many respects, so it is not that relevant to national security *currently*. That could change within a year's time. He is focused on the trajectory, and he treats the expectation as the same kind of speculation as Nvidia's valuation: a lot of smart money thinks it pays off soon. Cyber, for now, is not at the level of a malicious actor pulling off a devastating attack on the grid, though we should still prepare. Virology is the exception. Reasoning models have gotten very good at STEM PhD topics, including the literature, and they are rounding the corner on helping in practical wet-lab situations. That, he says, already has national-security implications, and only very recently.

Guo invests in defensive cybersecurity and biotech discovery and worries about a safety-versus-benefits tradeoff. Hendrycks does not buy a big tradeoff. For bio, if you want those capabilities open, talk to sales, get an enterprise account, keep a small refusal for unknown users. That is basically xAI's risk-management framework: do not expose expert-level capabilities to people you do not know; if you do know them, sure. The same move works for cyber. Once you have done that, you have basically taken care of malicious use behind your API, and that is about the best a company can do. Policymakers, he thinks, have not internalized any of this. They still think the labs are selling hype and that employees do not actually believe AGI could arrive in the next few years.

Asked how AI gets used as a weapon beyond virology and security, he would not expect a bioweapon from a state actor; from a non-state actor it makes more sense. Cyber makes sense from both. Then come drones, exotic EMP research, better drones, and situational awareness. The informational case is the one he wants people to sit with: an advance that lets you know where all the nuclear submarines are, or pinpoint hardened land-based launchers, could disrupt second-strike capabilities and mutually assured destruction. That is not even a weapon. It is just better information, and it could be extremely destabilizing. The default conventional AI weapon is still drones, and he thinks it would be a mistake if the U.S. were not trying harder to manufacture them. He has started working with an electronic-warfare company. Elad Gil adds that battlefield awareness often comes down to radio and radar, an area where AI is already relevant in Ukraine.

A Wall Street story is his picture of the next control problem. There used to be a human in the loop for every decision; later, before the requirement was removed, rows of people just clicked accept, accept, accept. Some AI contexts are getting to a similar state, and he would not be surprised if more of that decision-making gets automated. The question then is reliability, which is why reliability research is useful. Separately, he flags a real policy tradeoff that is not the API one: if you turn the pain dial all the way up on China in export controls, and if AI chips are the future currency of economic power, you increase the probability they want to invade Taiwan. They already want to. Denying chips *and* the latest semiconductor tools gives them all the more reason.

People hear risk management and think pause. Hendrycks's objection is geopolitical, not sentimental. If you slow down voluntarily, you make yourself less powerful and let worse actors get ahead. You can sign a treaty, but it would be imprudent to assume the treaty will be followed; you need a threat of force or a verification mechanism behind it. Absent that, a voluntary slowdown does not make sense. As a proxy, compliance with treaties and norms around cyberattacks and corporate espionage has been terrible. He uses that fact to hit two camps at once.

One camp is last year's *Situational Awareness* paper from Leopold Aschenbrenner, a safety person whose idea was to beat China to superintelligence as hard as possible. Hendrycks's critique is that the plan assumes corporate espionage will not be a thing. Gil notes that 30% plus of employees at top AI companies are Chinese nationals. Get rid of them and they go to China, where they may beat you because they are extremely important to U.S. success; keep them and you accept information-security exposure. Hendrycks's conclusion is that you live with both. Immigration policy for talented AI researchers should make it easier to stay, and that discussion should be kept totally separate from southern-border policy.

Racing in drones or AI chips is fine. Racing to a superintelligence weapon on the assumption that the other side will not steal it, copy it, or stop you is a tall claim. Unless you move researchers to the desert and lock down information security — which would take years, and would send some of your best scientists back to China — they can co-opt it. Even then, if they saw you pulling ahead toward a system that could crush them, they would not sit idly by and accept your dictates until the end of time. A trillion-dollar compute cluster in the desert, visible from space, has basically one plausible read: a bid for monopoly on superintelligence. The rational reply is to hit the data center first.

He reaches for nuclear history. There was a brief period when even pacifists such as Bertrand Russell argued for preventive destruction of the USSR. The opportunity window may never have really existed, but there was a prospect of it for a time. He does not think that window exists here, because of the United States' multinational talent dependence. We are clearly nowhere close to walling China off from insight or imitation. Powerful systems may arrive before a years-long securitization could even be finished. If there were a magic button that slowed everything 3x, he does not even have a position on whether that would mean fewer mess-ups. Given structural constraints between companies and states, a lot of the nicer risk-mitigation gestures are infeasible.

The alternative, proposed with Schmidt and Wang, is mutually assured AI malfunction — MAIM, a nod to MAD. In nuclear strategy, states deterred a first strike because they could retaliate; they had a shared vulnerability. Hendrycks expects something similar once AI is viewed as pivotal to a nation's future, once a country is on the verge of automating almost all AI research and bootstrapping to a superweapon that could crush others. At that point China says it will do something preemptive, a cyberattack on your data center; the U.S. might do that to China. Russia, coming out of Ukraine and noticing the U.S.-China AI gap — say later in a year when a big chunk of software engineering is being impacted — would keep tabs and threaten the same: if you try to crush us with this, we will prevent it.

Espionage and sabotage are cheap enough to make the threat credible. All they need is a zero-day on Slack, and they can know what DeepMind, OpenAI, xAI, and others are up to in high fidelity. Right now they would not bother, because the capabilities are not yet that destabilizing and a lot of decision-makers still are not taking AI that seriously. He thinks that changes as the systems get more powerful, and that the resulting deterrent is what keeps anyone from trying to build a weapon that lets one country wipe out the other — the scenario Aschenbrenner-style racing was flirting with. MAIM is for superweapon-class projects. Conventional military AI, drones included, he expects countries to keep racing on, the way they raced on bows and nuclear arms that it just made sense to threaten each other with.

The nuclear, chemical, and biological analogies he likes are dual-use and non-proliferation, not a total freeze. Chemical and biological weapons are a poor man's atom bomb, which is why rivals still signed the Chemical Weapons Convention and the Biological Weapons Convention: terrorists having those things benefits nobody. AI chips in the hands of Iran are the same kind of shared interest. That is coordination that is incentive-compatible. It is not an incentive for both sides to pause all AI development. It is a reason to deter particular forms of development that have a plausible prospect of a decisive edge.

Guo asks for a first step the current administration could actually take. Hendrycks's list is cheap, in the same spirit as the easy anti-terror refusals for companies. Stand up a CIA cell that spies on other states' AI programs so Washington is not caught by surprise. Have Cyber Command ready attacks that can disable foreign data centers if a destabilizing project is running. For non-proliferation to rogue actors, adjust export controls so the government reliably knows where AI chips are, for the same reason it wants to know where fissile material is. Allies notify you when chips move and get a license exemption; enforcement officers prioritize end-use checks. These are a few texts, or a basic document, away. "80/20 is a lot of it." Safety, he keeps repeating, is not really that much of a technical problem. It is a complex geopolitical problem with technical aspects.

DeepSeek's highly compute-efficient training on a much smaller scale of importable chips does not change his mind about compute security. It undermines Manhattan Project strategies that rely on restricting another superpower's ability to make models. You cannot robustly restrict their capabilities. You can restrict their *intent*, which is what deterrence does, and you can restrict rogue actors such as Iran, which is what export controls and compute security should actually facilitate. China will probably keep getting some fraction of the chips; the job is to know where they are. You could even coordinate with China to keep chips out of rogue hands. Tighten if you want, but if your biggest priority is that China gets none at all, they will steal the weights anyway.

He is not even sure the existing controls were tried. To his understanding, AI chips were not a substantial priority among leadership at the Bureau of Industry and Security. Did any enforcement officer go to Singapore to see where 10% of Nvidia's chips were going? They would have very quickly found they were going to China. A basic end-use check would have taken care of that. Non-proliferation of chemical agents and fissile material shows export controls can work if people care. Gil cannot see a world where another great power's leadership, believing there is economic value here, says they do not need the capability. Hendrycks agrees. The right mindset is nuclear strategy: stockpile, secure second strike, watch what they are doing, stop proliferation to rogue actors when capabilities are extremely hazardous. The Cuban Missile Crisis came pretty close to all-out nuclear war. It depends on what we do. Get a handle on the geopolitics and you are left with domestic problems such as automation.

In the last minutes Guo turns to evals, the other half of Hendrycks's public work. He has been making evaluations for about as long as he has been doing AI research: ImageNet-C before ChatGPT, then MMLU and a MATH dataset, and now Humanity's Last Exam, plus an eval Guo mentions called Enigma. HLE asked professors and researchers around the world to submit a really challenging question with a definitive closed-ended answer — the kind of problem they encounter in their own research. The point is to mark the end of the road for exam-like benchmarks. When performance on that set is near the ceiling, the genre of multiple-choice and simple short answer is roughly expired. That would be an indication you have something like a superhuman mathematician or STEM scientist, at least wherever closed-ended questions are useful, such as math.

It does not measure the other thing that matters: ability to perform open-ended tasks. That is agent evaluation, and it will take more time. The plan is to collect digital tasks, let a model work on them for a few hours, and see whether they get completed. A test like that is coming out soon. Knowledge in the academy can already look strong; agent skill is still near the floor. Models are still extremely defective as agents, though that could change overnight. The reason to keep building exams at all is public understanding. If every evaluation is saturated, it is difficult to have a conversation about the state of AI. Nobody knows where it is, where it is going, or the rate of improvement.

The intelligence frontier is jagged in ways that still surprise him. Models cannot fold clothes and can answer a lot of tough physics problems. It seems totally plausible they will be better than humans at mathematics not too long from now and still unable to book a flight. When they are better, humans can still verify some of it: run an AI mathematician's proof through a proof checker. Taste is harder to confirm, if that notion even makes philosophical sense. Overall he thinks we are on track for systems with really good oracle-like skills — you ask, and it says something insightful or pushes a bound of knowledge — without, for some while, being able to carry out tasks on people's behalf. That is why we still do not take them that seriously: they cannot do a lot of trivial stuff. When the agent skills arrive, there are not many barriers left to economic impact. The vibes shift. It becomes obvious this is in a category of its own, much bigger than the app store or social media.