Superhuman Go Has a Blind Spot. Balaji Bets on a War of the Gods

Balaji Srinivasan arrives on *The Cognitive Revolution* with a résumé that already spans several industries. He founded Teleport, Counsyl, and Earn.com, became CTO of Coinbase after the last of those sold, taught statistics at Stanford, and was a general partner at Andreessen Horowitz. He now writes and invests independently as the author of *The Network State*. Host Nathan Labenz, an entrepreneur and AI investor, wanted every one of those lenses pointed at a single question: what is happening with AI, and how does it turn out.

The case that sets the tone is a 2023 result from the safety group FAR AI. Researchers found an attack on a superhuman Go model using a strategy that would not work at all against a decent human player. A quality amateur could brush it aside. The superhuman system was simply blind to it. Nathan flags that he hopes to have FAR AI's Adam Gleave on the show to talk through the work. Balaji treats the result as more than a cute hack.

If open weights let you probe a model without even changing them, he conjectures, you can reverse-engineer adversarial input and break the system. That is almost a safety property. You get power words you can speak to make the thing power down, like movie monsters that cannot see you if you stand still. The model can be overwhelming on one axis and trivial to topple on another. For humans you cannot copy the brain and analyze it; with open weights, you can.

Balaji dates his snapshot to December 11, 2023, because the field moves too fast to speak in the timeless present. He splits today's breakthroughs into two overlapping schools. The DeepMind school is known for reinforcement learning: superhuman Go, pixel-level video games, protein folding, systems that get the basic rules and then figure the rest out. The OpenAI school is generative: large language models, diffusion models for images, and the multimodal stacks that glue them together. What they share, he says, is an assumption of static rules.

The rules of chess and Go do not change with time. They are discoverable, like laws of physics. English grammar and word meanings do drift, but over decades, not weeks, so training data from five years ago is still valuable and the same input roughly gives the same output. Facts change, the Queen of England dies, a king takes her place, but that is not the same as the rules themselves moving. Markets are the contrast case. Repeat a strategy and counterparties see it and trade against you. Time-varying, rule-varying systems are, in his view, the true frontier.

Nathan cashes that out in the language of mundane utility: AI is good at tasks and not yet good at whole jobs. If you can define what good looks like, it can do the small thing; if the job is handling whatever comes up, it is not there. Both expect some synthesis of hard-edged reinforcement learning and softer language models, the direction the then-current Q* rumors were pointing. Nathan recites timelines as a mantra. The first public instruction-following model arrived in January 2022; GPT-4 finished training a little over a year before they sat down. The integration into real systems is amazing and still a work in progress.

Adversarial examples are the hardest evidence for the boundary. Add a perturbation a person cannot see and a classifier calls a giraffe a dog. FAR AI moved the same idea into Go. Language models have a universal-jailbreak problem even when you only probe the weights. Balaji's intuition is that we can probably robustify these systems; if we cannot, they are fragile in a fundamental way, which would mean you can always kill them. Nathan knows of no architecture that is demonstrably robust. The conversation then turns to complements: state-space models beside transformers, control-theory stacks under generative front ends, Skydio-style drones that take verbal instructions and execute with deterministic low-level code. Unique, clean session data, Balaji notes, will still crush zero-shot hope in domains like Figma or DJI.

Asked whether AI favors incumbents or disruptors, Balaji's answer runs through politics. Competent CEOs in their prime, Amjad Masad at Replit, Dylan Field at Figma, will AIfy existing products, a pivot as large as desktop-to-mobile. The open question is whether disruption will be allowed in the U.S. regulatory environment. He analogizes the Biden administration's reporting threshold, training compute above 10 to the 26, to the apocryphal Bill Gates line that 640K of memory should be enough for anyone. In his view it is the first IRS form for a software FDA: the first form always targets only the super-rich, then a million rules grow on top. Slippery slope, he says, is not a fallacy. It is how things work.

Nuclear technology is the script he does not want repeated. Fear-driven regulation in the early 1970s bent nuclear costs from flat to vertical. ALARA, as low as reasonably achievable, meant you keep adding safety until the cost eats the value. The world got nuclear weapons and not the cheap power, nuclear ships, and other civilian uses that were on the table. Blow people up, cannot generate electricity: worst of all worlds. He worries San Francisco invents AI, lobbies for the rules that then choke it, and becomes the Xerox PARC of the era, the place that built the future other people harvested. The platform under the models is not just compute. It is law.

The world he sees is specifically tripolar, not unipolar, bipolar, or multipolar: NYT, CCP, BTC, or U.S. tech, China tech, and global tech. Apple sits on the U.S.-China boundary because it makes hardware in China; Binance sits on the China-global boundary. He guesses Musk's Grok, trained in Texas, may grow into red AI, while OpenAI and DeepMind stay blue AI, alongside Chinese AI and decentralized AI. China can mandate clean real-time datasets, pay the population in digital yuan to label data, and clone physical robots at scale, sidewalk delivery already common in Chinese cities. The decentralized pole's weapon is indifference to copyright: Sci-Hub and LibGen as a method, Hollywood itself born when immigrant filmmakers fled Edison's East Coast patents for Southern California. He and Flipkart's founder talked up a generic AI, servers in India, train on movies, books, and songs, and let the lawsuits run in the rearview while a domestic industry grows. Distance from San Francisco can be an advantage, the way Spotify was far enough from the labels that Napster, LimeWire, and Grooveshark were not.

Who staffs that third pole? He bets on Indians and Israelis, diasporas included, on the order of 30 to 50 percent of tech founders by his count, plus crypto wealth that does not bow to Washington or Beijing. The freedom stack is censorship-resistant social media, cryptocurrency, and open-source AI with no compute limits, the same spirit as The Pirate Bay and Bitcoin. Blue America, he argues, faces an economic apocalypse from the ABCs: AI hitting licensed, overpriced paper jobs, Bitcoin hitting monetary power, China hitting military primacy. The U.S. lawyer count went from under 100,000 in 1880 to a curve that went vertical after the 1970s. People who think the federal government will guarantee AI safety are, in a Leninist phrase he uses on purpose, useful idiots. The same state, he says, is credibly alleged to have funded risky coronavirus work in Wuhan and has risked hot war with Russia, with nobody punished. They do not care about AI safety. What they care about is AI control.

Nathan puts the fork on the table. Balaji's political analysis, he says, seems to assume AI stays a normal technology. If, over a decade, mixed architectures become as robust as we are, faster, able to do full jobs or even be tech CEOs, then realpolitik may not be the right frame. Lab leaders at OpenAI, Anthropic, and DeepMind talk as if systems more powerful than any individual human are the main question. Balaji does not dismiss that world. He wants mathematical, engineering, and physical constraints on the table first, work he says he is doing with Martin Casado, who sold Nicira for about a billion dollars, and Vijay Pande of Folding@home.

Math is the first cage. Chaos and turbulence make even simple systems unpredictably sensitive to initial conditions, so you can design problems an AI cannot forecast without actually running them; it has to react in real time. Cryptography is the other math bar. AI can fake an image; it cannot fake a digital signature unless it breaks factorization-class problems. He has a slide deck on the pairing: AI makes everything easy to fake, crypto makes it hard to fake again. On-chain timestamps are facts an AI cannot cheaply rewind. If someone proves P equals NP, some of this breaks. Within present mathematics, it is a cage.

Engineering is the second. A lot of AI-safety writing, including work he associates with Yudkowsky, strikes him as theoretical rather than empirical, and as Abrahamic rather than Dharmic or Sinic. The off switch exists. Apple and Google can push software updates to a billion devices; a few executives coordinating can hunt a virus. The closest thing to a self-sustaining program is not Stuxnet, it is Bitcoin, and Bitcoin lives because millions of humans keep powering nodes and maintaining code. Living off the land, in the next Turing-threshold sense, would mean controlling enough robots to mine ore, stand up data centers and generators, and defend them against humans, a literal Terminator scenario. His order of magnitude is hundreds of millions to a billion internet-connected autonomous robots. The current count is effectively zero. Sensors and actuators are the hole in the paper scenarios. He built robot systems for a genome-sequencing lab a decade ago and knows how much physical friction sits between a plan and a motion.

Hypnotizing humans is not like controlling drones. A hand lives or dies with you; an employee has a payoff function and will defect. One CEO and two employees already yield eight win-lose combinations; n people yield two to the n. OpenAI's board fight, in his telling, locked because equity pointed a thousand people at the same cell, an $80 billion company. Drones have no mind of their own. Humans would have to be paid and aligned so hard they die if the AI dies, an ant-colony setup we do not have. Nanotech gets the same cold water: Feynman and Robert Freitas published decades ago, artificial replicators have not arrived, and much of what is called nanotech is rebranded chemistry. DNA polymerase and ribosomes work because evolution built them. Super-viruses run into immune systems honed over billions of years. Pause-AI activism, taken far enough, looks to him like a step from Aum Shinrikyo: people who hear a voice and then act.

If those constraints hold, the picture is not one vengeful god. So much paperclip thinking, Balaji says, is Abrahamic furniture we do not notice: a wrathful deity who used to turn you into a pillar of salt and now turns you into paperclips. The polytheistic version is that everyone has their own AI gods, and then there is a war of the gods, Zeus and Hera rather than a single maximizer. India, he claims, is culturally more amenable to many gods than to one god and one state; China builds a civilizational root over its stack, the way it insisted on a Chinese version of social media and electric cars. Combine those and you get, at minimum, U.S., Chinese, and Indian varieties, coordinating humans who still have to tend them.

Nathan's Elon-Musk-in-a-loop worry, a cloneable first-principles mind dropped into an agent scaffold, is, in Balaji's reframe, not a separate AI Elon. It is amplified intelligence, a human-AI fusion, maybe another 20 or 30 Elons, which he thinks is within envelope. Harvard anthropologist Richard Wrangham's *Catching Fire* argument is the deep history: cooking moved metabolism outside the body and freed energy for brains; clothes saved us from growing fur; tools saved us from fangs and claws. Man-machine symbiosis is not a new thing. It is the old thing that broke humans away from primate lineages that did not use tools. No modern person survives without someone else's stove, tractor grain, heat, clothes, phone, or computer.

That history, he says, is also the next political axis. Uncle Ted is Ted Kaczynski, anarcho-primitivism, degrowth, back to the jungle. Uncle Fred is Friedrich Nietzsche, the Übermensch, acceleration, the stars. Left statists will call the second camp a crime against the state, right conservatives a crime against God; left libertarians will say it is my body, right libertarians it is my money. A hundred Uncle Teds in the woods would strip the trees. Elon Musk is already a fusion: without the internet he does not reach 150 million people. AI Jesus or AI Lee Kuan Yew can appear as a hologram and answer questions, but engineers still pour the ghee, the sysadmins of the temple. Until the Terminator robot count exists, it is still a human-AI symbiosis, and the priests still have the keys.

Nathan does not concede the loop. Proto-agents already wrap a model in plan, act, feedback, and repeat; the bottleneck looks less like architecture and more like the models not yet being good enough to get over humps. Habitat degradation is another path besides axe or drone: we did not hunt most species to death one by one, we changed the environment until they had no place. Balaji hears that as Hai, human-plus-AI, outcompeting other people economically, which he is willing to call a normal-world disruption, not the end of the world. The ABCs of apocalypse for blue America still sit on his side of that line.

Nathan stacks empirical points. A recent DeepMind medical paper asked models to do differential diagnosis on hard, rare-disease case studies. The AI beat both the human and the human-plus-AI condition; the human made the AI worse. Chess already walked this path: humans best, then hybrids best, now pure AI best. Medicine is not there yet, but the shape is visible. NVIDIA's Eureka, from Jim Fan's lab, had GPT-4 write reward functions for robot skills such as twirling a pencil, outperforming human experts. Training AI, on that evidence, is already something AI can do better than people. Balaji's reply is to keep asking for the actuator. Infinite cheap medical advice is a billing event getting hit, not a paperclipper. The smarter you are, the smarter the AI is; vocabulary finesse is the resolution of the amplifier. He would rather call it amplified intelligence than a replacement.

The most technical exhibit is model-to-model communication that skips human language. Salesforce's BLIP-2 froze a vision model and a language model and trained only a small adapter, a couple of days of GPU time, so an image could be discussed in language the LM had never been trained on. DeepMind's Flamingo is in the same family, a slightly Frankenstein assembly that worked. High-dimensional spaces appear bridgeable; insisting that logs stay in readable language is a tax, because language is a compressed channel. Nathan points to Ajeya Cotra's gradual picture: not necessarily a foom, but AI-centric production over a decade or a generation, executive language models over lower-level specialists, humans sliding toward the margin. Balaji steel-mans that as massive economic disruption he actually expects. The line he will not blur is disruption versus extinction, and extinction still needs an actuator: a self-radicalized human, which is conventional terrorism, or a private-key-holding drone fleet we do not have.

By the time those drones exist, Balaji expects cryptography to have grown into their brains. Crypto wallets, without meaning to, solved the public-key-infrastructure problem cryptographers nagged about in the 1990s and 2000s: a private key that is both secure and available. Hide it in a lockbox and you lose it; post it on a website and it is not secret. A wallet on your person is both, because losing either property loses the money, and hundreds of millions of people now hold such pairs. Zero-knowledge proofs make cryptography more programmable: prove you ran a given model without revealing weights, or compute a SNP distribution at one genomic locus without leaking sequences. Nathan had already done an episode with Daniel Kang on proving model execution. The rare consensus in the room is that the stronger AI gets, the more cryptography matters. AI is offense; crypto is defense.

Proliferation, on Balaji's side, is good, because if everybody has a model, nobody has a relative advantage. Nathan will not call it bad as a confident position; even much of the safety world, he says, now treats Llama 2 as useful for interpretability, though some opposed the release. The disagreement is about the next jump, systems powerful enough that naive extrapolation makes open release hard to predict. Balaji keeps dragging the conversation back to the same test. Businesses disrupted and MDs earning less is conventional. Skynet is not. Hypnotized humans look like a cult or a terror cell, and we have playbooks. A Starcraft-like fleet of robots is new, and by then private keys fragment the address space so not every drone is everybody's to command.

Nathan asks him to keep his identity small, in Paul Graham's phrase, and not run AI through a tribal prism. Gain-of-function is the analogy that worries him: the opposite of a couple of government labs should not be a wet-lab kit on every kitchen table, and compressing AI power into a home kit feels like that misstep. Balaji grants that A being bad does not make B good. His claim is that asking the U.S. and Chinese governments to police this is the fox guarding the henhouse, the same actors credibly tied to a novel pathogen, unpunished, using the crisis to gain power. Secular people, he says, have replaced God with gov, a female Democratic version with infinite money and a Republican version with the U.S. military, both praying to a dead god that cannot clean San Francisco's streets, that spent a hundred billion dollars on a California train without laying a track, that now pays more in interest than on defense. Passing a global compute limit requires guys with guns on the other side to enforce it. Other tribes are already funding their own gods. Polytheistic AI is not a metaphor he will put in the back of the analysis. It is the plot line everything else is already joining.

He leaves a honest gap. Maybe someday every civilization that makes it can crowdfund and operate its own AI brain, and the gods go to war. That day still requires AI that can live without humans. Bitcoin cannot. Until then the practical stance is the one he has been circling: treat the model as an amplifier, put the facts you cannot afford to fake under signatures and chain timestamps, and stand in the jobs where the rules keep moving. Labenz ran out of time first, after more than two hours, and asked him back.