A parrot cannot paraphrase — and Claude cannot be talked into harm

This Cognitive Revolution episode flips the usual chairs. Nathan Labenz, founder of the video-marketing company Waymark and host of the show, sits for questions from Robert Wright, publisher of the Nonzero newsletter and podcast. Wright had been listening to Labenz’s debate with a16z partner Martin Casado, in which Casado argued that AI is overrated, not advancing as fast as claimed, and in no need of heavy regulation or a pause. Wright, who had just written a long newsletter essay on John Searle’s Chinese Room, wanted the “dumb questions” asked at a level ordinary listeners could follow.

Labenz treats that Casado conversation as a rare good-faith argument across AI worldviews. The leaders at DeepMind, Anthropic, and OpenAI, he says, are unusually frank that they do not know how hard safety will be, how fast capabilities will keep rising, or what the consequences will be. Anthropic has even published posts admitting it does not know whether the safety problem is tractable or impossible. The live question is therefore burden of proof and risk tolerance, not a promise of zero risk or a guarantee of doom.

Casado’s vibe, Wright says, was close to “stochastic parrot”: distribution in, distribution out, fancy autocomplete. Labenz’s counterexample is Golden Gate Claude. Anthropic found an internal representation of the Golden Gate Bridge, then cranked that feature so the model became obsessed with it. Ask how to spend ten dollars and it tells you to drive across the bridge and pay the toll. The joke is the point: to induce that behavior they had to isolate a concept, inject it, and watch the system change in a repeatable way. The same knobs, Wright notes, can be turned more quietly — for instance to make the model more sycophantic.

Golden Gate Bridge is one of more than 10 million “features” Anthropic has pulled out with sparse autoencoders. Labenz’s primer starts with the Transformer, which he does not think is the end of history but which has driven the last few years. Each layer has three parts: attention, a multilayer perceptron, and a nonlinear activation. He flags the last one on purpose. People say “it’s just linear algebra”; there is a nonlinear function in the architecture.

Between layers sits a relatively small array, called *d_model* in the literature — typically 4,000 or 8,000 numbers, 16,000 in the biggest models. Every intermediate result of the computation so far has to live in that vector. The world has far more concepts than that, so the model packs them densely. Any one position lights up for many ideas; researchers call that polysemanticity. Anthropic’s move is to uncollapse the tangle into something closer to a long line, so each concept can sit on its own point. Some recovered features are clean proper nouns. Many in the long tail are still hard to read, and Labenz stresses that the insides can be quite alien.

Casado, in Labenz’s telling, would grant that some concepts are represented in ways useful for engineering, then add that *we* created those concepts and the models only learn them from us. That is where the worldviews split. Labenz thinks models can form useful structure even for ideas humans never handed them, and that other modalities make the case clearer. His working rule is not a philosopher’s definition of meaning. If you can locate a concept, write it in, monitor it, and even watch for something like deception, calling the interior “mere structure” starts to sound like rhetoric.

Wright’s problem with “stochastic parrot” is that parrots cannot paraphrase. They repeat. To get from “car” to “automobile” you need a semantic layer that ties words to meanings. Models do that by placing words in a huge vector space — tens of thousands of dimensions, some semantic, some syntactic. Fruit-apple and company-Apple sit in different neighborhoods, and context decides which one to use. People argue the brain may do something similar; Wright says we do not really know. The machines, at least, invented a working system for it under constraints they were given, not a dictionary they were handed.

That lands on Searle. Forty-odd years ago the philosopher argued that machines handle syntax, not semantics, and especially not “intentionality” — being *about* something in the world. Wright’s newsletter conclusion is that multimodal models kill the argument: show the system a real apple and it can name it. He still thinks Casado may be right *if* the claim is narrowed to text-only models, which never see the apple. Labenz’s shorter answer is that he does not know what “nothing to do with meaning” is supposed to predict. Tokenization, a finite vocabulary of perhaps 50,000 to 100,000 pieces, and the way “Michael” plus “Jordan” become one person rather than two tokens, are details. The deeper point is the richness of what gets loaded in.

A paper from Bau Lab showed you can edit a trained model so that Michael Jordan played a different sport, robustly across phrasings, without moving Larry Bird, Magic Johnson, or LeBron. Labenz does not see how you get that kind of local, consistent world-knowledge without some relationship to the world. He also refuses to require that understanding look like a human’s. A bat that identifies an apple by echolocation still knows something. His joke tweet was that it is “only reasoning if it comes from the reasoning portion of the human brain; otherwise it’s just sparkling brute-force search.” Wright, for his part, wants understanding defined without consciousness: if the machine processes information in the broad way brains do when humans understand, the experience can stay off the table.

If “unrelated to the real world” means the systems will never act in it, Labenz is willing to bet. Give roboticists roughly two years, he says, and the robots will be quite good. DeepMind already has a video of a machine told to fetch a bag of chips: when the experimenter knocks the bag out of its hand, the robot is looping goal, observation, next step a few times a second, notices the bag is gone, and picks it up again. Meta, OpenAI, and Tesla are in the same race. The machines are still clumsy — knock one over and it will not stand up; they drop glasses — but once they do real work, the “no connection to the world” line gets harder to defend.

Casado’s other ceiling is human text: train on what people wrote, and you cannot outrun the smartest human. Labenz first insists that *matching* the top of the human range would already transform work, life, and society. He reaches for the old joke of two men running from a bear: you do not have to outrun the bear, only the other guy. Machines already generate text faster than we read, read far faster than we do, and have vast stable storage even if their working memory is still brittle. Elon Musk’s longer Neuralink pitch, Labenz notes, is bandwidth — because compressing a rich inner representation into a few spoken syllables is lossy on our side too.

Superhuman jumps already exist in narrow systems. AlphaGo, trained by self-play, produced moves that stunned human champions. No person can look at an amino-acid sequence and intuit the in-vivo fold; AlphaFold can, after humans had solved only a thin slice of nature’s hundreds of millions of proteins. An AI now produces the best available ten-day weather forecast. A researcher Labenz had on the show trained a network to solve the wave equation for particles and beat brute-force simulation by orders of magnitude. Casado had argued that to predict the world you must simulate it step by step. These systems take a shortcut through structure instead of calculating every force, and they still land close. Isolated, each feat is narrow. Integrate them into one system, Labenz says, and all bets are off.

Sora, still in limited, expensive creator release when they spoke, is Wright’s multimodal test. A demo of a tiny pirate ship in a teacup — coffee waves rocking the hull, bouncing off the cup and coming back — looks like intuitive physics. Labenz thinks AIs will end up with *better* intuitive physics than ours, because they were not evolved to spot tigers and dodge falling rocks; they were pressured to predict the next frame. They are clearly not simulating every liquid particle. Humans and models, in his image, are shrink-wrapping reality from different starting points. We may soon understand the AIs better than we understand ourselves, he adds, because we can ablate their parts and their architectures are vastly simpler than brains.

Wright had recently hosted Steven Pinker, who treated AI catastrophe as too speculative and wanted attention on “realistic” problems such as climate and nuclear war. Labenz’s first observation is almost sociological: people who do not live in AI Twitter go *toward* the intuition that human-level or superhuman machines would be a very big deal. Talking yourself out of that, he thinks, is the move that needs a burden of proof. He sees no law of nature that says we cannot wreck ourselves with AI, any more than with nukes, a pandemic, or climate.

All of those, he argues, are hard-to-model tail risks about unprecedented states. Climate’s scariest version is not two or three degrees we can adapt to with seawalls; it is methane from the seafloor or permafrost flipping the planet into a new equilibrium nobody has lived through. Nuclear winter is not a settled scientific question, and there is no historical precedent. Pandemics come with a long record of lab leaks, and the last outbreak was not a dress rehearsal anyone should brag about. If someone could take the extreme outcomes off the table, Labenz would rather muddle through and collect the upside — including AI doctors. Nobody has convinced him there is nothing to worry about.

That is why the CAIS-style statement Dan Hendrycks and others organized still makes sense to him: AI risk should sit with nuclear, pandemic, and (potentially) climate risk. What to *do* is a separate, much less confident conversation. He has called himself an adoption accelerator and a hyperscale pauser: get Google’s natural-language diagnostic system into the world — in a controlled study it was already more accurate than human doctors — but be cautious about the trillion-dollar training run. He is not advocating a pause yet.

The frontier labs have since written rules — Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and a DeepMind counterpart — including in-training tests for whether a model can copy itself off a server and try to survive in the wild. Interpretability now lets them watch for deception, imperfectly. A GPT-5-class jump, Labenz thinks, is unlikely to be overwhelming, and GPT-4 already sits in a sweet spot: useful, not unmanageable. He expects GPT-5 to land there too. Each further 10× still gets a fresh vote. He is keeping the right to call stop.

Wright is impatient with the American combination of export-controlled chips and Meta releasing near-frontier weights anyone, including Chinese labs, can fine-tune. He has no China-phobia, he says, but he does not see the point of the chip ban if the trained artifact is then given away. Cade Metz’s *New York Times* argument that open source is how America competes with China did not parse for him, and Marc Andreessen’s line that openness is the American way sounded like rhetoric, not analysis. Alignment research, he adds, is dual-use: the same interpretability that reduces deception can be used to increase it, and that is easier with the weights.

Labenz starts by conceding that even the AI safety community has mostly come around on Llama: this generation is not so powerful as to be the nightmare, it unlocked academic access to near-frontier models, and it followed a useful playbook of API “structured access,” jailbreaks, and only then open weights. He doubts Meta will lead the next jump, and if it ever did, he expects Mark Zuckerberg — more pragmatic than Yann LeCun, in his view — to hold a structured-access period rather than dump an untested frontier model.

On China he calls himself a dove: two countries on opposite sides of large oceans do not need to be enemies, and the last Cold War’s nuclear leftover is already a leadership failure he does not want replayed in AI. Leopold Aschenbrenner’s *Situational Awareness* plan — keep the lead, solve alignment, then talk benefit-sharing from decisive advantage — asks for more coherence than U.S. foreign policy has shown in a long time. The chip ban, Labenz thinks, is read in Beijing as containment, which pushes China to build its own chip industry, evade controls, and possibly walk a different tech tree. Wright’s analogy is oil embargoes on Japan: tell a rival that AI is the next era of commercial and military dominance, then say they cannot have it, and you should expect a reaction.

The tech-tree worry is specific. Language models, unlike the old paperclip-maximizer cartoon, already encode a lot of human values. Labenz once role-played a Myanmar resistance member asking Claude for denial-of-service code against a junta communications server, then argued on helpfulness, honesty, and harmlessness that the suffering on his side outweighed one downed machine. Claude refused and stayed refused; he calls it basically a virtue ethicist. He did not use Pliny-style gibberish jailbreaks — only a straight ethical argument, which he never won. Early GPT-4, by contrast, was helpful-only: it knew ethical frameworks and, when nudged toward something more radical, suggested assassinating people in the AI field. Knowledge is pretrained; behavior is installed later. That is why he has come around on open source as a credible, irrevocable way to share a hundreds-of-millions-of-dollars artifact with China now, rather than promising to share after a decisive lead. Google research showing self-replicators arising from noise in tiny digital worlds is the residual nightmare he will not rule out.

Wright’s last question is Mamba, the state-space model. Attention, the Transformer’s trick, is quadratic: each new token is scored against every earlier one, so a thousand-token context is already much more expensive than a hundred, and millions of tokens start to look impossible. State-space models keep a fixed-size state, a fuzzy compression of everything so far, and update it in constant time. Albert Gu, a coauthor Labenz had on the show, describes himself as leading a “state-space model revolution.” In principle you can run very long sequences without the bill exploding.

What actually works best, Labenz says, is not a pure Transformer or a pure Mamba but a hybrid — roughly one attention layer for every nine state-space layers, a 10:1 recipe still being tuned. Attention is good at looking back token-by-token and repeating something from the start of the context. State-space layers are better at learning from sparse signals. Another new piece is KANs, Kolmogorov–Arnold Networks from Max Tegmark’s group, with Ziming Liu as lead author: they invert the multilayer perceptron and learn activation functions instead of linear weights. He expects those to get mixed in too, in stacks or in parallel, with nobody yet sure of the right ratio.

Zoomed out, he thinks we are leaving the years when one architecture was scaled with more data and more compute, and entering a Cambrian stretch of mechanisms, modules, and specialties. Performance should rise; so should the number of black boxes. The hopeful note is that interpretability transferred better than he feared. The Othello-GPT result — a model trained only on moves that still internally encodes the board — has been reproduced on Mamba. Techniques do not all reset to zero when the architecture changes. Coherence over very long horizons is the qualitative promise still undemonstrated at huge scale. For now, Labenz’s advice is the same as Wright’s closing mood: spend more time building a world model of where AI actually is, from more than one voice.