Sleepwalking Into an AI Arms Race: The Case for a Different China Strategy

Wright, who publishes the Nonzero newsletter, opens by explaining the piece that started the conversation: "AI surrenders to the military industrial complex." The news hook was that Anthropic and Meta had decided to do business with the US military, including through Palantir, a defense contractor long associated with predictive rather than generative AI. Nathan Labenz, host of The Cognitive Revolution, had just come from a Berkeley conference called the Curve, where he joined an AI war game.

Wright says the surprise was the interesting part. Anthropic has a reputation for safety-consciousness, and some people inferred from that a general dovishness or leftish politics. Wright, who grew up in a military family and respects the institution, says the safety mindset never implied that.

The clue, he argues, was in Dario Amodei's long essay "Machines of Loving Grace." Most of it lays out positive futures, but a short section sets out Amodei's foreign-policy outlook for AI. In Wright's reading it adopts the Washington framing that democracies are in an existential contest with autocracies and embraces the chip war on China, a framing Wright thinks risks becoming a self-fulfilling prophecy.

Labenz urges everyone to read Amodei's essay, whose first half he calls one of the best recent statements of AI's upside. Amodei, with a background in biology, imagines a "compressed century" in which 100 years of biological and medical progress arrive in perhaps five to ten years. Labenz says he has been hoping for AI lab leaders to offer this kind of positive vision.

The geopolitics part, he says, felt like the tech mind assuming a simple fix. Democracies cooperate, AI gives them an edge, adversaries are isolated, and eventually those adversaries take the deal. The phrase "decisive strategic advantage" captures the idea of becoming so far ahead that the other side has no choice but to accept terms nobody has specified.

Amodei's timeline for what he calls powerful AI is roughly two to four years, with 2027 the most likely year. That is also the year the war game simulated. Both hosts note that this would put the decisions in the hands of current leaders, Trump and Xi, which Labenz finds concerning even as he concedes the alternatives are poor.

Asked why China hawkishness has become dominant, Labenz admits he has no scholar's theory and is relaying hearsay. One explanation he has heard is that in a polarized capital, the only bipartisan way to move anything is to frame it as getting ahead of China. Bridges, roads and supply chains all get pitched that way.

He has mixed feelings. Subsidizing domestic chip manufacturing seems prudent to him, since the US cannot currently make advanced chips at home. But he thinks broad decoupling from China makes future conflict easier and more likely, and he would rather see economic ties continue.

The result, in his impression, is a political environment where people feel the anti-China train is the only one running, so everything else gets hitched to it. Wright says he worries about how quickly this hawkishness has spread, including among AI people who are just learning national security.

Wright outlines the scenario from Leopold Aschenbrenner's "Situational Awareness" essay. A country that reaches artificial general intelligence first could gain overwhelming dominance within months, perhaps by instructing the AI to do things like undermining a rival's social media. Wright says the essay is long, yet it leaves its fundamental assumptions unstated.

Labenz says the Curve was meant as a mixing bowl of AI-safety people and national-security people, with no single worldview dominating. Almost everyone took strikingly plausible rapid progress seriously, and he recalls reading about GPT-2 while his now five-year-old son was about to be born. He was struck that large majorities seemed to support export controls, though he does not see why that follows.

On how fast takeoff happens he sees a wide range of views. Some expect superintelligence soon after general intelligence through self-improvement loops, while others, including Amodei in his reading, expect limits from the need to run real experiments on scarce GPUs. Labenz thinks reality-based feedback, as in game-playing AIs that learned from win-or-lose signals, will likely let models become superhuman at coding and experiment design.

The conversation turns to claims that AI is hitting a wall. Wright wonders whether the chain-of-thought approach in OpenAI's o1, a model effectively debating itself, resembles the self-play that powered game AIs. Labenz says that is directionally right, and notes that a Chinese company, DeepSeek, shipped a comparable reasoning model only about nine weeks after o1 appeared.

On whether China stole the technique, Labenz says IP theft cannot be ruled out. But he thinks the more likely explanation is simply that China has many strong researchers who were probably already on this path. He believes o1 is still a cut above on benchmarks and likely more so beyond them.

He also relays the widely accepted story of the OpenAI board crisis a year earlier: that a reasoning breakthrough and long-standing doubts about trust in Sam Altman combined to spook Ilya Sutskever. He stresses the details may be wrong, and the lesson he draws is that once a capability is demonstrated, rivals are strongly invited to reproduce it.

Wright returns to Ilya Sutskever's hope that nations would see AI as a shared challenge worth governing together. He complains that China hawks rarely test that idea or count the costs of an arms race: lower transparency, a chaotic environment that rewards skipping safety testing, and the destabilization AI will cause in the US. He adds that authoritarian China may have more incentive to regulate AI and could prove more stable.

His central worry is a game-theory one. Until recently both sides depended on high-performance chips from TSMC's Taiwan fabs, and a war would incapacitate them, a kind of mutually assured destruction. Now that the US and its allies cut China off from advanced chips and the equipment to make them, China cares less whether those fabs survive, and is also building its own capacity.

Wright says he rarely hears this scenario in the foreign-policy establishment and credits analyst Paul Triolo with writing about it. Labenz says it does come up in AI circles, though in a minority, since the dominant story is to hold China down while the US moves ahead. He adds that if China believed Western timelines for powerful AI, it could try to stop US access to chips.

Labenz describes a conversation with Zvi Mowshowitz, who argued that TSMC is still standing because China does not believe in an imminent takeoff in its gut. Chip manufacturing equipment is also extremely delicate, with Dutch ASML teams on site at TSMC, so incapacitating a fab would be easy. The implication is that if China comes to believe the Western timelines, it could act differently.

Labenz recommends following Miles Brundage, who recently left OpenAI and argues that the West must show China its AI development is benign. He adds the hard part: it is difficult to do when it may not be benign. The US president says in print that America must win the AI arms race, and lab leaders say the same.

He notes that Sam Altman used to say AI decisions should not be driven by what China might do, and has since flipped. Altman has also written that the world will have AI with Western values or autocratic values, with no third way. Both hosts think Beijing is likely to read all this as a threat to the regime, and Wright suggests the stated plan of backing China into a corner with terms unspecified will be hard for it to accept.

Wright reads Amodei's line about isolating adversaries until they "give up competing with democracies" and asks what that means. He argues Silicon Valley libertarians fear that a Chinese-led world would impose its values on others. His rebuttal is that there is little evidence China tries to export its system. It wants partners in its economic zone and UN votes, and is more transactional than the US, which does try to remake countries in its image.

Labenz agrees this is common and likens it to a hall of mirrors: each side expects the other to hold it down, so adversarial action looks baked in. He mentions Joshua Steinman, a first-term Trump National Security Council member, who called this "the physics of the environment." Labenz thinks that is overstated, because there is no law of physics preventing us from breaking out of a prisoner's dilemma.

He points to signs of hope: the Biden-Xi statement that AI will not be put in the nuclear chain of command, and AI safety people inside China. There is an underground meme that Xi, being a scientist, might de-escalate. A former national-security official at the Curve could offer no answer on how China will respond, saying only that Trump likes deals and might cut China into one.

Wright admits a better approach is owed, and presses Labenz on what an AI deal could look like. Labenz says the technology itself shapes the answer. A big item from the Curve was unpublished alignment research that, in his summary, shows none of the current AI control techniques is robust, so there is no credible plan for controlling a super-powerful AI.

He says safety groups, Anthropic especially, have been smart to invest in "scary demos," findings vivid enough to show the public or even the Chinese government that racing is dangerous. If the world comes to see that nobody knows how to solve the problem, he can imagine something like a jointly run island in the Pacific where researchers from both sides work with an on-site reactor and data center.

The obstacle is verification. Data centers will keep being built, nobody can tell from outside what they are used for, and China has more visibility into the West than the reverse. Even if both sides agreed the situation is scary, each would doubt the other was secretly pressing ahead, so the game theory is hard.

Wright approaches from the other end: put the US-China relationship on a fundamentally different basis. History shows that when each side sees the other as an adversary, defensive moves look offensive and threat and inflation feed each other. He says Vietnam and Iraq were mistakes because neither was an actual threat to the US, and argues that with enough trust, including inspectors visiting data centers and regular data exchange, an AI arrangement is conceivable.

To get there, he says, the US should abandon the habit of demanding that other countries become like us. He thinks human-rights-based sanctions are almost always counterproductive and hurt those they aim to help. On Taiwan, leaders should ask for assurances: China promises no invasion or aggressive displays, and the US sends no more weapons and discourages provocative visits.

Wright notes that individual political incentives often block the logical path of two powers cooperating. American politicians gain by scaring voters, and Sam Altman can win subsidies by doing so. In his view the unreliable player in the relationship is often the US, because its internal dynamics make it so, and he says that if China refuses, at least the attempt was made.

Labenz says he will happily wear the "China dove" label while agreeing on human-rights concerns and having no sympathy for communism. He suggests separating the Chinese ruling elite from the Chinese people and 5,000 years of civilization, and warns against leaping from criticizing current policy to condemning "Chinese values." He admits that scientific exchange did not prevent the First World War but still wants Western and Chinese AI leaders and safety researchers to talk more.

He notes that AI now helps break the language barrier, and floats ideas such as sister-city programs. He also thinks banning TikTok would be a mistake, since it gives a window into Chinese culture and could be done later if a hot war occurred. He says it is far likelier that AI brings a terrible outcome than that his grandkids speak Chinese.

Wright adds that this carries a risk of being labeled a sympathizer, and says his upcoming book argues for cognitive empathy, meaning simply trying to see another's perspective. He recalls living near the Pentagon as a child during the Cuban Missile Crisis and sees the same dynamic of stigma toward understanding the other side. The one difference, he says, is that today's information environment is less opaque. Labenz does not want the answer to what he did during an AI arms race to be "nothing."

Labenz describes the four-hour exercise at the Curve. It starts in 2027 with an AI that can do routine knowledge work and moves into a takeoff in which the AIs improve themselves. Players represented the president, Congress, OpenAI, other developers, and the AI itself, played by Anton. Labenz played the safety teams, the public and the press, and before the game the AI's alignment was secretly determined by dice rolls.

As the safety teams, he felt no test could justify racing ahead, so he demanded artificial speed limits and threatened to blow the whistle. When the OpenAI player refused, citing China, he went to Congress, which formed a committee that did little. Public alarm grew, and he imagined protests and even data-center bombings, though none of it changed the outcome.

A US-China war followed, with the players' own estimates of who would win roughly 50-50. Ships were sunk, chip fabs went offline, and the American president handed military control to the AI, which delivered a decisive victory. China took a face-saving peace rather than going nuclear, and when the cards were flipped the AI was benign.

Afterwards Wright questions how central a threshold-and-takeoff assumption is. Labenz says fast takeoff is not necessarily critical to the Amodei or Aschenbrenner scenarios. He then describes his "tale of the cognitive tape" slide, where AI has caught up with human experts on depth of knowledge and reasoning but still lags in robustness to trickery and knowing when to "break frame," and expects those gaps to close.