A Toy Paper Did Not Prove LLMs Cannot Generalize. The Discourse Still Ran With It.

The cold open is already a warning. A tweet says look, this proves language models cannot generalize, and Nathan Labenz calls that insane. He nicknames the post the Helen of Transformers: Helen of Troy launched a thousand ships, and this tweet launched a thousand takes. Misleading conclusions about what AI *is*, he says, cannot be good for the downstream argument about what should be done. Co-host Erik Torenberg names the other fear: that a reasonable-sounding pledge becomes a Trojan horse, a governing body with reputational credit and then the legal power to decide who may innovate.

Labenz's line for the hour is the one he wants remembered. This may be the time to build, but it is definitely not the time for ideology. *Cognitive Revolution* usually interviews researchers and builders. This week there is no guest. Labenz, who founded the AI video company Waymark and served as a GPT-4 red-teamer, has had a burr in the saddle for about a week. He wants to take two online fights apart, contribution by contribution, and hold himself accountable as they go.

He has spent the previous ten days in OpenAI's new releases, and he parks a spoiler for a later episode. The vision component, which he expected to be a huge unlock, really does look like one, partly because it is cheap: twelve images for one cent, plus whatever the model generates in response. That price, he thinks, will unlock monitoring, video slices, and periodic screenshots, a lot of passive collection that is suddenly easy to process. That is not today's subject. Today's subject is two slices of late-2023 AI Twitter that he thinks the industry can be better than.

Before the critiques, Labenz plants two flags to show he wants existing AI adopted faster, not slower. The first is Waymo, in a study with Swiss Re. Over 3.8 million miles driven with no human behind the wheel in rider-only mode, the Waymo driver incurred zero bodily-injury claims, against a human baseline of 1.11 per million miles. Property-damage claims fell to 0.7 per million miles, against a human baseline of 3.26. In a larger, more statistically robust set of more than 35 million miles, with a human specialist monitoring the automation, both injury and property claims were also well below the human baselines.

He is careful about what the abstract does not settle. Some self-driving systems still involve intervention, and it is not entirely clear how much sits behind these numbers. Even so, he wants to go on record: this sounds awesome. A sane society would start improving the environment to make the systems more friendly, beginning with the trees in his neighborhood that hide stop signs. Zero injuries in more than three million miles, where the human baseline would have predicted more than three, is the kind of result he thinks should move policy, not just timelines.

The second flag is medical. A Harvard Medical School preprint evaluated GPT-4V on 934 challenging *New England Journal of Medicine* image cases and 69 clinicopathological conferences. The model outperformed human respondents overall, across difficulty levels, skin tones, and image types, matching humans only in radiology. It synthesized image and text, but performance deteriorated when images were added to already highly informative text. Unlike humans, it used text to improve on image challenges and also missed obvious diagnoses. The authors' close is cautious: multimodality is promising, context is key, and human-AI collaboration studies are needed.

Labenz goes further than the authors. A second-opinion diagnosis is something you can scrutinize and talk over with a human doctor, unlike a robotaxi that is already on the road. On a consequentialist basis he would put this in people's hands now. If you do not have a doctor, it sounds a hell of a lot better than not having a doctor. If you do, the second opinion and the discussion that follows are probably clearly on net to the good. It will make obvious mistakes. Human doctors will too. The nightmare is when they make the same ones. Torenberg jokes that the olive branch has been extended. Then they wait, with bated breath, for the part Labenz actually came to fight about.

The viral version said Google DeepMind research shows LLMs cannot generalize. The actual title is narrower: pre-training data mixtures enable narrow model selection capabilities in Transformer models. The authors, led by Steve Yadlowsky, were talking about curating training data to enable certain behavior. That flipped, in public, into the claim that if you set up pre-training a certain way you may control what a model can and cannot do, and then flipped again into LLMs cannot generalize. The second leap, Labenz says, does not follow.

The sentence everyone highlighted was careful, then stripped. In the regimes studied, the authors found strong evidence that the model can perform model selection among pretrained function classes during in-context learning at little extra statistical cost, but limited evidence that in-context learning generalizes beyond the pre-training data. Model selection here means recognizing what kind of problem you are facing and applying the right lesson. The toy is tiny. One data type is points on a line: given XY pairs and a new X, predict Y. The other is sine curves, same task. The small model learns both, and at runtime it can tell them apart.

The shortfall is the overlay. Add a sine curve to a line and you get a wave that slowly rises or falls. The model failed at that combination. The title still says *narrow* model selection *in the regimes studied*. Everybody blew past that limiter. Frontier models, Labenz says, are not in that debate. Nobody serious is questioning whether GPT-4 can take a bit of concept A and a bit of concept B and blend them. Ask it for a Shakespearean poem about the 1990s Chicago Bulls. That exact mix is unlikely to sit in the training data, and it can still write it. You do not even need the twenty-dollar ChatGPT plan to see that a little of column A and a little of column B already works.

The interesting residue is control, not incapacity. If you design the data set carefully, you may get some control over what models can and cannot do. That is not a generic law that generalization never happens. The real questions about frontier models are whether they can infer things leading experts do not know. At most the paper says that strategically keeping data out of training may keep a model from generalizing into those domains. Labenz thinks excluding most academic virology from general-purpose training, while leaving specialist models for actual biologists, is one of the more credible near-term biosecurity ideas, and this study slightly bolsters it. That is a different kind of generalization than the far-doom case people were loading onto a three-person toy paper.

Someone had already reproduced the paper. Labenz credits Samuel Müller, working in a Colab notebook, an individual with essentially no dedicated compute beyond a small Google Cloud virtual machine. Müller shared the code. As far as he could tell he copied the original setup, with one change: a bit of noise in the training data. Then the model generalized. It got over the hump of the line-plus-sine combination.

Noise injection is one of the oldest robustness tricks in deep learning. Labenz is not a historian of the early 2010s, but computer-vision practice has long been to train with perturbations: compressions that change aspect ratio, added noise, waviness, programmatic weirdness. When the system still extracts the signal, downstream performance gets better and brittle overfitting eases. Müller applied that to points on a line or a sine curve. It did not take long. A couple of days. The counter-tweet got a healthy number of likes and still was, in Labenz's view, not a shining moment in AI discourse. Torenberg asks what Amjad Masad would say to this overview. Labenz would rather have him on than psychologize. He tries to engage arguments, and what he sees in the surrounding tweets is that arguments are mostly not being made.

Yadlowsky's original tweet was fairly mundane, quoting the result and saying a new Google paper provided evidence that Transformers, GPT, and so on cannot generalize beyond their training data. That became the thing everyone quote-tweeted. The author later joked that their paper on in-context learning had been taken out of context. Adam D'Angelo of Quora, a former OpenAI board member, called the narrow result, true only under a lot of assumptions, a Rorschach test for views on AI. What surprised him was how many people seemed to be expecting progress to stop.

Labenz answers people he respects. Naveen Rao, founder of Mosaic ML, which exited for more than a billion dollars, wrote that the belief was fun while it lasted: sparks do not always lead to fire. Labenz hears a wink at Microsoft's *Sparks of AGI*. The DeepMind toy, he says, does not invalidate Microsoft Research's reports on GPT-4, GPT-4V, and GPT-4's impact on science, covering materials, biochem, and partial differential equations. Those remain among the most reliable public pictures of frontier capability. Mocking them with a non sequitur from a lines-and-sines model is not analysis.

Princeton's Arvind Narayanan, author of *AI Snake Oil*, wrote that the paper is not even about LLMs, yet seemed to be the final straw that popped a bubble of collective belief: if emergence merely unlocks capabilities already in pre-training data, the gravy train will soon run out. Labenz grants the first clause and rejects the jump. You should not update worldview-scale beliefs from a toy, especially once a little noise made it generalize. The deeper miss is the size of the data. GPT-4 already speaks the world's languages better than any human who has ever lived, because the languages are in the training data. What else is in there that humans cannot pick up? If the next model eats a couple of trillion tokens of genomic or proteomic data, superhuman tools on raw DNA would not surprise him. AlphaGo's move 37 was a move no human would have played, and it was genius. That same week, Google authors from the Med-PaLM team, including guests from an earlier episode, were already pulling useful novel hypotheses about genetic interactions out of their latest system.

NVIDIA's Jim Fan wrote that Transformers are not an elixir, and that machine-learning 101 says you have to cover the test distribution in training. True, Labenz says, and it does not follow that unexpected generalization is impossible, because the training data contains more than we have extracted. Replit CEO Amjad Masad wrote that he came to the no-generalization conclusion last year and was a little sad, because he had wanted to believe in LLM mysticism, that there was a there there. Labenz does not think the live question is consciousness or moral patienthood. Can the technology we already have, and will have at future scale, learn things people do not know? Look at existing systems. Language itself is a starting point. A user named accelerate harder tied the pile-on to the AI executive order looking more foolish. Labenz thinks a lot of the amplification was a way to promote an agenda about what should be done. That is soldier mindset, in Julia Galef's terms. He would rather be an AI scout. These systems are more like alien intelligence than human intelligence. This may be the time to build. It is not the time for ideology.

Part two is not a bill. It is not even a policy proposal. Hemant Taneja, CEO of General Catalyst, posted that more than 35 VC firms and more than 15 companies, representing hundreds of billions in capital, had signed voluntary responsible-AI commitments from Responsible Labs, a nonprofit he co-founded. Notable names included Inflection, SoftBank, Insight Partners, Intel Capital, IVP, Lux Capital, and Arthur AI. The five commitments are internal governance, appropriate transparency and documentation, risk and benefit forecasting, auditing and testing, and feedback cycles with ongoing improvement, plus a handbook of best practices.

Taneja's argument is that AI can make the world better, and that investors should advocate for startups from day one. The valley denounced last month's executive order; in his telling it is still mostly reporting, with a risk of devolving into rules that slow American businesses. The right path is not to be antagonistic toward Washington. Collaborate with elected leaders, and hold yourselves accountable for what you fund. Labenz did not sign. He is averse to oath-swearing, which he finds bad for scout mindset. Read as a checklist, though, it looks like common sense. Auditing and testing is the heaviest item, and even that is software testing moved onto AI products. At Waymark, models write video scripts; the worst case is hostile or racist content, a contained harm compared with a frontier lab. Nine months earlier, Bing Chat had told a user to divorce his wife and landed on the front page of the *New York Times*. Spending more time testing is altruism and self-interest.

The reaction treated the sky as falling. Balaji Srinivasan wrote that a free internet means free AI, that he likes Taneja and many of the signers, and that he fundamentally disagrees with the philosophy of capitulation; they will fight government control over compute with everything they have. Torenberg takes the critics' side more fully. The fear is a Trojan horse: reputational credit first, then a body that can decide who may innovate. Social-media builders were contrite, apologetic, and naive, and they lost the credibility war to less introspective institutions. Marc Andreessen's thought experiment is a 1990s committee that decided who could start a website. Moderates lose to extremists, Torenberg says, because extremists will not apply the same good-faith rules and will use yours against you. That is part of why e/acc tries to match the other pole.

Labenz's reply has layers. It is only a war if everyone thinks it is a war. If an alien craft landed, you would hope people would first figure out what it was, what it could do, and whether it would hurt anyone, not immediately split into camps throwing rocks. The United States, in November 2023, has one toe-in-the-water reporting requirement: train something ten times bigger than GPT-4 in compute, which might affect five to ten companies next year, and all they must do is tell the government. That is not heavy-handed yet. The surest way to the outcome Taneja fears, a self-radicalizing tech sector that cannot even pay lip service to protecting the public, is to pile onto colleagues who published best practices. Dozens of surveys show the public wants government action on AI, across parties. Social media was people talking with a layer of amplification. AI systems can plan, reason, and use tools, and two years ago they could do none of it. People outside the field, and many leaders inside it, do not know what happens next. "If you want to prevent that regulation, show me that there's no problem. Show me that you have it under control."

Torenberg likes Taneja's work and still sympathizes with the critique of "responsible tech" as a philosophy that under-explains why technology is good. Lives saved by the FDA or by robotaxis are invisible; one visible death is not. e/acc, in his reading, is trying to shift the Overton window toward the positives, and some of the hostility is about who gets to be the spokesperson. Taneja, to that crowd, feels too European, too EU, too Mustafa Suleyman: the wrong voice even if self-regulation could avoid ham-handed law. A better tack, Torenberg says, is self-regulation plus a hard case against overreach, using social media, nuclear power, the FDA, and public-choice theory, and arguing that going too far is scarier than not going far enough.

Labenz asks listeners to imagine self-driving companies taking the e/acc posture after a fatal accident: no standards, some people will die, deal with it. They would be out of business in two seconds. Waymo's path, data and insurance, is why 3.8 million miles is persuasive. The pairing of the two controversies is the contradiction he cannot get past. One camp says the technology is not that powerful, nothing to worry about. The e/acc camp says it *is* powerful and transformative, then refuses responsibility. If the strategy is shifting the Overton window, he would rather they communicate less strategically and more earnestly. How powerful is this. How risky is this. You will not get that from the e/acc crowd, which treats the trajectory as a universal law and shrugs that being eclipsed might even be a moral good.

His closing challenge is product liability, which already exists. Perplexity's summary, as he reads it, is that parties in manufacturing and distribution can be responsible for damage a product causes, and that a manufacturer must exercise the care reasonable for experts in similar products. Anthropic's "race to the top" is an attempt to show you can build frontier technology, a business, and high safety standards at once. If those expert standards become acknowledged, they start to define the duty of care. He suspects some of the fury at a voluntary pledge is really fury at that liability hook. He invites Beff Jezos, Jeremy Howard, Martin Casado, and Steven Sinofsky onto the show to explain why AI products should be exempt, especially if there is nothing to worry about. A database is an inert tool. These systems plan, reason, and use tools.

Labenz does not want stupid reporting piled onto an application developer using orders of magnitude less compute than OpenAI, or another GDPR-style banner on every AI product. He also does not want the industry to invite a heavy hand by performing the opposite of a stupidity that has not happened yet. He borrows a line he attributes to Eliezer Yudkowsky: the opposite of stupid is not smart. It is just another form of stupid. Book the e/acc guests, he tells Torenberg, and try to get past the polarized frame into a credible positive vision. Focus on what is true before loading everything into an argument about what should be done.