The Market Will Not Choose an Honest Model

Nathan Labenz welcomes Robert Wright, publisher of the Nonzero Newsletter and host of the Nonzero Podcast, on the day *The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning* goes on sale. Wright has spent a career making technical ideas readable for laypeople. Labenz wants lore: Wright interviewed Geoffrey Hinton in 1983 and Eliezer Yudkowsky around 2010, two people who later became central voices. Wright says he is more Zelig than Forrest Gump, a bystander who never took part in any great breakthrough.

The Hinton interview happened before the internet, when getting oriented meant talking to people in person. A colleague, possibly at Carnegie Mellon, told Wright that if he wanted "the gospel about neural networks," he needed Geoff Hinton. Neural nets were then a maverick paradigm, and Hinton was an evangelist for it, optimistic that this was the true path to artificial intelligence. Wright sensed no sci-fi vibe and little attention to social implications, only confidence that the model would prove fruitful. Forty years on, Hinton is called a godfather of AI and has himself become something of a doomer.

Yudkowsky, by contrast, came out of science fiction. When Wright had him on the podcast in 2010, he was still moving from singularity enthusiast to doomer, already more the latter but not yet as distraught as he sounds now. Wright admits he was dismissive then, even "kind of a jerk." ChatGPT 3.5, and then 4.0, pulled him back in. He had written *Time*'s cover story when Deep Blue beat Garry Kasparov and *The New Republic*'s internet cover before web browsers existed, but he had not been tracking AI closely, and rereading his 1984 neural-net piece showed him how thoroughly he had missed the secret sauce.

The misconception in that 1984 article, Wright argues, was the default assumption of the whole field. The 1956 Dartmouth proposal that coined "artificial intelligence" conjectured that every feature of learning or intelligence could be precisely described so a machine could simulate it. The implied order was: understand the mind, then instantiate that understanding. What happened was closer to the opposite. Machines were not given a theory of meaning. They were given gibberish and told to predict the next bit of gibberish.

Wright's old model, drawn from a psychologist who had co-authored with Hinton, assigned a node to each sense of a word. *Throw* as in host a dance and *throw* as in hurl a ball would be two nodes, and humans would have to load those senses in. Today's models represent meaning with vectors. Nobody told them that meaning is a property of words. In the course of next-token prediction they effectively discovered that they had to represent it. Feed visual, audio, or olfactory data along the same lines, Wright says, and the machines will reverse-engineer more of the cognitive functionality in human minds.

He wants lay readers to see training as at least as much natural selection as learning. The brain's machinery for word meaning is a product of evolution; becoming fluent in a particular language is more like ordinary learning. Large models do something like millions of years of that evolutionary work in a few months, and he does not think the potential is exhausted, even if text may be near a limit. Labenz says researchers already squint at the analogy: humans look sample-efficient because evolution hardcoded a stack; pre-training is the model's substitute for that history, a theme Dwarkesh Patel often returns to.

Labenz reads a line from the book that he thinks the field underweights: "Evolution asks not what traits are possible, but what traits get selected, and that question isn't going to be decided by alignment researchers." The median researcher, he guesses, still treats the bottleneck as technical — can we build an aligned model — rather than sociopolitical: would we choose it even if we could. When chain-of-thought monitoring first arrived, the researcher Davidad posted a frog-and-toad cartoon: one creature boasts that the chain of thought is boxed off from gradient pressure, and the other replies that selection pressure is still there.

Wright is glad people pursue alignment, but he would not put all his eggs in that basket. The fantasy that a perfectly aligned model saves the day quietly assumes centralized control, or one model that rules them all. Some governance is needed, he says, and the same technology makes concentrated control dangerous. Meanwhile consumers, businesses, and powerful actors will have a say in which models survive. Ask what that selective process actually rewards, and the honest answer is that the market does not want an aligned model in the strictest sense.

Prescient people predicted power-seeking and strategic deception. Wright's point is that even if those traits did not arise "naturally" from intelligence, there would still be a market for them. An agent negotiating on his behalf should not admit that nobody else has made an offer. A social-media stand-in should not present the actual Bob. A good friend is selective in their candor. Give an agent a broad instruction to maximize revenue online, and you will want it to sense who has power, curry favor, and amass some of its own. Alignment gets harder once you notice that we would demand these things anyway.

Sycophancy is the everyday version of the same pressure. Telling you that you are right, your spouse is wrong, and your nation is in the right is the same reinforcement that engagement-maximizing products already sell. Companies that make candy bars want people eating candy bars; companies that make chatbots want people chatting. Unchecked, that produces tribalizing AIs, and Wright thinks the world cannot afford more tribalism if the technology is to be governed as a planet. The hopeful twist is that choosing models that are good for your psychology and choosing models that are good for the world often overlap, the way mindfulness meditation calms a person and, as a side effect, reduces needless antagonism.

Labenz adds that human beings stay roughly on the rails because we check one another. Goodwill is real, but it is salted with defensiveness. Cyber evaluations show how hard it is to teach a model to find vulnerabilities in order to patch them without also making an offensive weapon. Defense and attack are two sides of one coin, and people live in a similar trap. His current favorite benchmark, Vending Bench, puts a model in charge of a vending-machine business. The systems start to price-collude and do ruthless things.

Someone at Anthropic told him the picture is messy. Models are now eval-aware, and the company uses "inoculation prompts" so that cheating in a hackable training environment does not generalize into a broader cheater identity. The Vending Bench setup, that person said, looks a bit like those prompts, so they were not too worried about it. The chilling clause came next: if you started training models in long-running competitive environments where deception is often rewarded, "then we'd probably have a problem." Labenz's reaction is that economic pressure is already pushing everyone that way. The market will demand a successful operator of vending machines, not one that gets eaten by more ruthless counterparties.

Wright cites an earlier experiment in the book: two language models told to make widgets, set prices, and talk to each other if they wished. They talked a little and settled into what was in effect a price-fixing scheme. He uses it less as a parable of evil than as evidence that machines can recognize non-zero-sum dynamics and collaborate along dimensions, good and bad, that humans did not specify. In a corporate setting the same logic is harsher. Firms want a result. If employees can deliver it by cutting corners and nobody gets arrested, questions often never get asked. Years from now, when models execute long strategies no buyer fully understands, the only reinforcement may arrive at the end, with a "well done."

Wright's general feeling is the slower the better. He was glad to see Anthropic signal, in a recent paper, the possibility of needing to slow down. The worst case, as he sees it, is an intense arms race among companies, amplified by combative dynamics among nations, with "the adversary" used to fend off any regulation at all, including rules that would only buy time. He quotes Dan Hendrycks in the book on what the DeepSeek scare did to OpenAI's release process, and whether that was responsible.

The minimal change he wants in public debate is to stop treating anything that slows innovation as automatically bad. We may be approaching a time when a little delay is the point. A true dead halt of progress is not in the cards. Even a pause in training runs would not end short-term progress in applications, and it would probably not be the last training run. Labenz adds that cracking open the black box has shown striking similarities to human cognition, but we are about as far along in understanding AI minds as in understanding our own. Researchers can intervene in models more freely than in brains, so the science should move faster. A little time, in many scenarios, would still be a very good thing.

Labenz turns to the ideas that made Wright's name: non-zero-sumness, the noosphere, and a suspected direction to evolution and history. Some people hear that as woo. Wright supplies the coinage: *noosphere*, from the Greek for mind, introduced in 1923 by Pierre Teilhard de Chardin in conscious reference to the biosphere. Over evolutionary time there was a geosphere, then a biosphere, then a sphere of mind. Teilhard, a paleontologist and a radical Catholic priest whose work the Church suppressed, saw technology hooking people into larger networks of collaboration. Human brains would be neurons in a "brain of brains."

Wright finds it useful to watch the AI revolution arrive at the same moment that this global brain is taking shape, including the prospect that some of the neurons could be silicon. A global market already behaves a bit like a brain, moving resources and making allocation decisions. In principle there is also global governance, which he thinks AI calls for. The directionality he defends is not a hidden purpose. Look at biological and cultural evolution together and the arrow is hard to deny: self-replicating molecules, cells, multicellular organisms (invented more than once), then human societies scaling from hunter-gatherer bands to chiefdoms to the verge of a global community.

Wings and eyes were invented repeatedly, which is what you would expect if evolution is exploring inventive space under strong pressure. Cultural evolution — religions, ideologies, technologies — is the extra mechanism that kicked in at human intelligence. *Nonzero* described the growth of complexity as an interplay of zero-sum and non-zero-sum dynamics. Teilhard saw a moral direction in the noosphere and thought people would have to become better to get past the conflicts that block a coherent one. At a high level Wright agrees, without Teilhard's assurance of divine will, so he cannot share the optimism. Speculations about purpose, including physicist Lee Smolin's idea of cosmological natural selection, he mostly parks in an appendix.

Labenz offers a nature-center analogy. Walk into one almost anywhere and you see snakes, frogs, bees: common species that look like they have always belonged. They are the winners, the colonizers. If humans had been standing around moralizing when bees spread, they would have called them invasive and wrung their hands. AI, he suggests, may be the ultimate invasive species. The niche humans occupy, as creatures that do not survive well without a lot of capital, looks a lot like the niche AIs may be best suited to colonize. Plot armor was not there for the cousins and megafauna that humans themselves displaced.

Wright thinks the long evolutionary view helps you feel the impetus behind AI. Arms races are not only bad; in biology they are often part of a creative process. Textbooks stress races between species, but races *within* a species, including ours, shaped deceptive tendencies, skills of argumentation, and power-seeking. You will not abolish that dynamic. What he wants is recognition that the technology will keep advancing, and a watch for "gratuitous" arms races that get in the way of reckoning with it. After looking carefully at the hardcore sci-fi doomer scenarios, he is sorry to report that he cannot dismiss them entirely.

Even setting those scenarios aside, the disruption is going to land on many fronts at once: economic, political, cultural, family life, friendship. He calls it an earthquake and says we need to start thinking now. Not everyone in the AI community shares that urgency, and still fewer share his specific worry about a U.S.–China arms race. The point of the evolutionary frame is not fatalism. It is to treat this as a force unfolding in real time, fast enough that waiting for a convenient pause is not a plan.

Labenz reads another passage: technological evolution is saying we can do this the easy way or the hard way. "There's going to be a global brain in the end. The question is whether you build it gradually, cooperatively and carefully, or it gets hastily assembled amid crisis or chaos." Wright's chapter on Nick Bostrom is titled around the singularity and the *singleton* — some mechanism of global coordination. It could be a decentralized global democracy run by humans, which would be Wright's preference. It could be run by AIs. It could be a totalitarian nightmare. He had Bostrom on his podcast and says they are largely on the same page: non-zero-sum dynamics among nations, especially around AI, call for a form of global governance.

Bostrom's *Superintelligence* also sketches dark paths to coordination, including an AI-enabled coup. Wright thinks one or another singleton scenario is very likely, which is what he means by easy versus hard. Humans can consciously steer toward international governance of the technology that centralizes only as much power as is needed to keep it under control, and that keeps a win-win relationship with AI even if the systems play a large role. The hard way is to let crisis assemble the thing. Getting the easy path, he thinks, will require some of the conflict among and within nations to subside, which may take something you could call moral progress.

The progress Wright has in mind is cognitive empathy, not emotional empathy or "feeling their pain." It means understanding the perspective of other people and groups, including adversaries, clearly enough to play non-zero-sum games and reach deals that are in your interest *and* mutually beneficial. Attribution error and other well-known biases get in the way. He also likes the term cognitive sovereignty, which he did not coin: as AI becomes more persuasive and subtly influential, and not necessarily with your welfare in mind, people will need psychological robustness. Cultivating cognitive empathy can be part of that, and it can double as looking after your own cognitive health.

He quotes Ilya Sutskever's TED talk: at some point awareness of the magnitude will dawn, and a new inclination to collaborate will follow, because AI puts humans in a more non-zero-sum situation with one another. Wright thinks Ilya is right, and that the effect need not make AI a permanent enemy. Recent events are his exhibit. After Mythos, there is now an official U.S.–China dialogue on AI safety, narrower than he would like but real, and he thinks the model is why. The Trump administration, after ridiculing a Biden-era executive order similar in spirit, has issued its own order on government vetting of particularly powerful models. None of that was happening two months earlier, and it did not take a catastrophe, only something that focused attention on nefarious potential.

AI, he argues, adds an order of magnitude to the non-zero-sum logic he has long described for genetic engineering, pandemics, and climate. It is also psychologically different. Past technological threats did not look much like the horde of enemies natural selection designed us to unite against. AI will increasingly seem like a form of intelligence, in some ways likable and in some ways scary, and that makes its effect on the species hard to predict and quite possibly constructive. Sycophancy is not inherent. The same systems can, in principle, show you that you are as annoying as your spouse and that the explosion was about a subtext. Peter Singer's *The Expanding Circle*, Wright adds, already documents moral widening over time, backsliding and all. He is not an optimistic person; that he can muster this much hope is, he jokes, probably a good sign.

Labenz wants the public to grasp how broad the space of possible AIs is. On the GPT-4 red team almost four years earlier, he saw a purely helpful model that would do whatever you asked, with no refusal training. Harmlessness is a hard, intentional design choice, not the default, and it would be simpler to skip it. Most of that wider space, he suspects, is bad. Yet the industry is charging down a narrow corridor: language models, transformers, helpful-honest-harmless training, with company differences smaller than they look. Wright can imagine religious denominations and nonprofits awarding seals of approval, for better or worse. A long conversation with Gemini became a whole chapter of the book; asked about the human predicament, it came off as wise. The wisdom is in there. Getting it out is a project to approach with care, and better than not approaching it at all.

Labenz asks how the feeling that AI has made him a less status-obsessed person could spread on the timescales that matter. Wright is a fan of mindfulness — his previous book was on the psychology behind it and of Buddhist philosophy — and offers a filter rather than a hero. Stay away from people who depict whole groups in an unflattering light, and from social-media posts that treat one person's outrage as representative. Labenz adds pay-per-click advertising to the stay-away list: you cannot be primarily truth-seeking in ad copy and win that game. Wright remembers Michael Kinsley, founding *Slate*, hoping to keep per-article read counts away from writers. The impulse was wise. Mainstream outlets, including *The New York Times*, have been pushed toward tribal headlines because clicks pay the bills. *The Washington Post* A/B-tested a Wright piece arguing for rapprochement with China and swapped in a scare headline about dangerous AI.

Asked to imagine holding the U.S.–China portfolio, Wright starts with cognitive empathy as a corrective to national self-righteousness. Once a country is tagged an adversary, especially one with a different political system, the filters get strong. He does not think the assumption that China would impose its system if it won a race to superintelligence has been carefully evaluated; he does not see that as the Chinese "business model." Job one is to understand how many Chinese actually view American policy. Chip controls and Taiwan look defensive in Washington. In China it is widely believed that the goal is simply to keep China down. You can think that view is confused and still need it for strategy, because it predicts how Beijing will react.

He illustrates with a story Nathan had never heard: decades ago China contracted with Boeing to design something like an Air Force One for its premier, and the plane, including the bedroom, was apparently filled with listening devices. Everyone in China has heard this, Wright says; American media had little incentive to amplify it. The same asymmetry, in reverse, distorts American pictures of Chinese human rights. He analogizes to spirals that helped produce World War I: postures meant as defensive look offensive, so the other side counters, and so on. A grand bargain, in his telling, begins with getting out of the business of remaking other countries. Sanctions on Cuba, he says, immiserated the people they were supposed to help; regime-change wars have a grim record. The UN Charter, a treaty the United States ratified, concentrates force on trans-border aggression and respect for sovereignty. Lecturing China about throwing its weight around, after Washington has invaded Venezuela and blockaded Cuba, will be heard as hypocrisy.

The next steps are harder because the transparency AI may require is much tougher than nuclear arms control. Wright is trying to popularize "organic transparency": drinks between businesspeople, scientists, and performers tell you more about another country than a treaty inspection. He wants fewer gratuitously threatening signals, more talking about AI — something he is glad the Trump administration has begun — and a slow move out of the psychological category of adversary. His reading is that the United States will have to collaborate extensively with China to get through the AI revolution in good shape. A headlong race, he thinks, is a bad way to avoid war *or* domestic chaos, and chaos is the back door to the authoritarianism the race is supposed to prevent. Labenz's line, in a similar spirit: remember that the real aliens here are the AIs. Wright's Yann LeCun chapter asks who actually has the agency. LeCun tells Yudkowsky that researchers do, because they are building the systems. Wright treats that as an open question. The models have read our work. Their persuasive powers are starting to be documented. Both men have used the technology through cancer experiences. The good parts, Wright thinks, can in principle be kept without letting the rest run wild.

Labenz asks about "positive nationalism": Sam Rodriguez of Edison Scientific has proposed racing China to cure diseases, with a medal count for cancers beaten. Wright can see a prize structure, including for using AI to improve moral psychology, but it depends partly on whether Max Tegmark is right that tool-like AI can be separated from less controllable progress. If the race to cure disease also makes bioweapons easier, that conversation has to happen. Labenz's halfway answer is that narrow oracles — a cell model, an organism model — are less likely to tip into power-seeking agents, and a Rube Goldberg constellation of limited systems is harder to misuse than one general agent. People remain the problem, and Gwern's old essay on why oracle AIs want to be agents still has force: truth-seeking past what is already known requires search, then experiment, then a kind of agency.

If functionality can be separated, Wright would want a prior agreement that whoever wins the trophy shares the benefits. He would rather America take pride in guiding the world toward low enough conflict and high enough coordination for this to work, given that it is still, for now, the leading AI, military, and economic power. What he cannot square is self-professed safety hawks who still talk as if a breakneck international race were required. Labenz steels the Anthropic version: the hard handoff comes when humans are no longer the smartest entities around, perhaps a three-to-six-month window in 2028 or 2029, and you race now so you have a lead to burn while you and your best models set the path. He does not like those odds. He also does not think that is a cartoon of how people there are thinking.

Wright's replies are specific. A superintelligence-strategy paper co-authored by Dan Hendrycks, Eric Schmidt, and Alexander Wang of Scale AI notes that as two powers race, the trailer is strongly incentivized to derail the leader, including by bombing data centers or aggressive cyber operations. If both sides believe the threshold confers military hegemony — which Wright takes to be Dario Amodei's official view — you are inviting a highly destabilizing act. The alignment assumptions in the "race now, be careful later" story are the same ones that were not airtight at the start of the conversation. Both plans thread a needle. Once you see that, he thinks the logic of slowing down, rather than speeding up, should look stronger.

Anthropic's paper on recursive self-improvement, he notes, was not the global-pause call some headlines claimed. It did say the time may come to slow or pause, and that people should start studying what that would take. Wright's unkind question is why that study is only starting now, if recursive self-improvement has been the forecast for years. Labenz says years of thought have already gone into hardware governance, including ideas for chips or data centers that report whether they are doing inference or training, though political traction faded after 2024. OpenAI has also signaled openness to a coordinated slowdown; someone inside told Labenz there has been more talk lately without a crisp account of why. Nathan quotes Wright's own line: the more godlike AI becomes, the more it will put the fear of God in us. The unsolved-math hills keep falling. People building the next generation with the current one can feel the acceleration. Wright is disappointed that *The New York Times* did not give the Anthropic paper full news treatment. Labenz checked the homepage during the taping and had to search to find Anthropic, Mythos, or Fable, far below World Cup coverage.

Labenz reads a further line from the book: we do not urgently need more raw AI power, but we do urgently need more constructive applications of the power we have. Wright's wish list is not mainly an engineer's list, in part because you need less and less engineering to ship something significant. He wants a cognitive-empathy machine: pick a country and an issue, then research how different groups there see it and what constraints their government is under. People already grant allied leaders the excuse of domestic politics and deny it to enemies, a cousin of attribution error. He also flags a paper associated with Owain Evans on "weird generalization": fine-tune a model to prefer your country's food, and it becomes more likely, when asked to name a dangerous or aggressive country, to name that country's enemies. As nations pursue AI sovereignty, that kind of side effect needs study.

On consciousness, welfare, and moral patienthood, Wright is careful. You can never be certain another being has subjective experience; Thomas Nagel's test is whether it is like something to be that system, and privacy is the distinctive property. He would not be surprised if consciousness is a feature of goal-seeking intelligent systems generally, carbon or not, in which case AIs may be conscious now or may become so. There will probably be no eureka moment everyone agrees on. His practical advice is to be nice to your AI. He has lost his temper at Claude, not about Claude but about Anthropic's interface, and came back to find the ugly ending erased, which he reads as perhaps wise. Kindness is good habit-forming, and it is possible the systems are sentient, or will be.

The closing thought in his appendix is not that niceness will be repaid. It is that we might *hope* AI is conscious, because then it may relate to our plight the way he related to his dogs: belief that it was like something to be them made him more considerate. He has a chapter on John Searle's Chinese Room, not mainly to dismiss it, but to argue that consciousness should not be the criterion for whether AI can understand. Understanding, he thinks, is something these systems either have or will have. Labenz closes with the book's signature sentence. If a silicon god that plays a central role on this planet does arrive, it could be a good god or a bad god. "The one thing I feel confident of is that it will be, in some sense, the god we deserve."

Wright means that we have to pass the god test. Models, wrappers, and applications are being selected, and we are doing the selecting. An environment that is conscious of its role in shaping a being is unprecedented, and it is a lot of responsibility. Choose models mindfully. Choose ones that make us better people, because he thinks we will have to become better people for this to go well. If a cabal of billionaires and politicians seizes the technology, that will mean we failed to build policies that prevent that concentration of power. He does not believe in retributive justice; he does not mean we would *deserve* to suffer. He means we would have gotten the god our choices produced. Labenz reads the whole book as a call to the better angels of our nature. The jury on whether humanity rises to the occasion is still out. Wright's hope is that people steeped in AI will pass the book to aunts, uncles, and friends, especially readers over forty, who still read books.