AGI Has Been a Moving Goalpost for Seventy Years

Nathan Labenz had planned a short worldview warmup before getting to open source and early AI regulation. He and Martin Casado, a general partner at Andreessen Horowitz, instead spent the first hour on a more basic question: how powerful will these systems actually become? Labenz, who focuses on what current models can already do, cites a formulation from Dwarkesh Patel: token for token the models already look as smart as most people, and what they still lack is packaging—long-term memory, autonomy, the ability to get back up after a fall. If that packaging arrives, the argument goes, a drop-in knowledge worker is not far behind.

Casado, who says he has spent three years looking at AI companies full-time and that a16z may have one of the largest such portfolios, answers from history. Over the past eighty years, he argues, so-called AI has made steady progress through winters and summers alike, on economics and on problem-solving. Every time a problem is cracked, people announce that AGI is at hand; then it turns out to be one modality, the definition moves, and the work continues. “This whole AGI thing has been a moving goalpost for 70 years,” he says. He hopes the tools keep helping humanity “stave off this universe that seems pretty destined on killing us,” but he does not see a step-change in the nature of computers or software.

His concrete analogy is self-driving. When he arrived at Stanford for a computer-science PhD around 2003, Sebastian Thrun had just won the DARPA Grand Challenge, driving a van fully autonomously. Everyone treated it as the threshold moment: robotaxi solved, a new era of vision and perception. Twenty years and nearly a hundred billion dollars later, Casado says Waymo-style unit economics are still about three times worse than Uber. Chess, handwriting, medical diagnosis, and mathematics have all seen machines beat humans for a long time, he notes, without yielding a general, economically viable solution for everything. What he sees in production is narrower: the marginal cost of content going to zero, and computers forming emotional connections with people—not a general knowledge worker.

Labenz pushes on Tesla Full Self-Driving, which already felt safer to him than many human drivers. Casado grants that specialized driving stacks can beat people in many cases, and he thinks that is a huge positive. He does not think you can take the system that is good at driving and make it a virtual girlfriend. The pattern he expects instead is that once AI is good enough, people change the environment rather than making the model more general: plant strawberries in rows with optical cues the robot can see; fix ambiguous highway signs; add lanes and markers. That is systems engineering, not cracked generality.

Casado’s deeper objection is statistical. Draw a Google search at random after de-duplication, and it is likely unique; count raw volume and 90% of queries look common. Networking, he notes, is famously the same way. In a heavy-tailed world the exception is the norm once you stop counting repeats. Building a general system therefore means that the majority of *new* things it must handle are exceptions, even though most of what it does day to day is routine. Self-driving is his Exhibit A: an essentially two-dimensional vision problem of streets and signs, nearly a hundred billion dollars in, still not economically positive at full autonomy.

He tells Labenz to diligence the classic IT-helpdesk pitch. A founder says the bot answers 90% of tickets; it sounds as if 90% of the labor vanishes. Look closer and 85% of that 90% is password resets, which a rules engine could have done. On unique intents, humans still handle about half. That, Casado says, was the chatbot industry for a decade, and the current LLM conversation is the same script with new lines. Models are good at predicting what a mean person would do next—kernel smoothing over positional embeddings, in his phrase—which is useful for the non-unique 80%. It is not a theory of the tail.

Asked to picture 2027, when an assistant is told to book a flight and hotel from a calendar invite, he expects another self-driving arc. Agentic systems will get 80% there; then people will change websites, add indicators, and build lanes the agents can follow. It will look more autonomous, but because software engineering evolved, not because generality was solved. Scaling laws that demand ten times more compute for the next increment will make that environmental work look cheap compared with brute force. Labenz agrees that today’s best systems close in on expert performance on *routine* tasks, and that the word routine is doing a lot of work.

Labenz’s hard case is Evo, the Arc Institute’s early DNA foundation model: seven billion parameters and 300 billion tokens, on the order of 2% of Llama 3’s training, doing next-base-pair prediction on bacteria and phage sequence. In gene-essentiality scoring, researchers perturb a gene and watch the model’s downstream confidence; if it becomes radically uncertain, the gene was load-bearing. Nobody taught it genetics. Casado’s reply comes from computational physics at Lawrence Livermore, including the nuclear-weapons program, and from protein folding at IBM’s Watson lab on Blue Gene in 1999, when the belief was that the forces were known and only compute was missing. His Occam’s razor is distribution in, distribution out: in a constrained axiomatic domain you learn a predictive distribution, and calling that emergent does not make it more than prediction.

Language, in this frame, is not the universe. Humans spent thousands of years staring at a fractal, heavy-tailed, nonlinear world and compressing it into words—“tree,” “rock,” “family”—a lossy linearization that makes structure predictable. An LLM that finds structure in that corpus is doing something close to compression. Casado cites a paper, then on arXiv, showing that how well gzip compresses a text corpus tracks how accurately a language model handles it. That, to him, is almost everything you need to know: the model exploits structure in the text, which “has nothing to do with underlying meaning.” Going from recovered text structure back to the universe “simply has not been demonstrated.”

Labenz counters with interpretability. Sparse autoencoders can isolate some thirty million features in a model such as Claude; jack up the Golden Gate Bridge direction and the system will talk about little else. Casado’s reply is that structure is not meaning. “Tree” is an arbitrary, useful human cut—one aspen grove or a cartoon drawing might both count—and models will recover those abstractions because the structure is sitting in the training text. Compression would do that too. Recovering a human concept is not the same as acting on the world, or inventing the next concept from raw experience.

He offers three mental models he actually trusts. Transformers, he says, are basically kernel smoothing, so error grows as data gets rare. In-context learning, in work by Columbia’s Vishal Misra, maps onto Bayesian updating: the prompt is new evidence that moves the posterior, and twenty years of Bayesian convergence results then apply. None of that forbids “new” outputs—interpolation and Bayesian reasoning produce novelty—but it bounds them. What he finds missing from the public debate is a theory of out-of-distribution generalization, as opposed to a game of anecdotal whack-a-mole that he compares to intelligent-design arguments. The hard human step, he adds, is looking at the universe and deciding that this is a rock and that is dust. He puts Llama-class post-training in the $100–200 million range, about half of training spend, which to him is the market admitting that supervised human abstraction is the scarce input. “No LLM that I know can do that. Not even close.”

They return to biology as a wager. Over the next year, Labenz expects scaled foundation models—now with eukaryotic sequence, expression, and proteome data the first Evo paper never used—to infer gene–protein–cell interactions too messy to simulate, then to yield medicines and disease patterns via counterfactual tweaks validated in the wet lab. That, to him, would count as looking at raw data and learning meaningful abstractions. Casado agrees it will happen. He has implemented Navier–Stokes for turbulent flow; industrial design already builds equations of state from empirical material tests and then simulates car crashes. Computers exist because ballistics are nonlinear: ENIAC was about 5,000 times faster than human “computers” filling logarithm tables. A biology model, in this telling, is the next instrument in that eighty-year lineage, not a new kind of general reasoner.

He adds a footnote from the last time simulation panic met export control. Working in the nuclear-weapons program at Livermore, he watched people worry that if Saddam Hussein got PlayStations he could cluster them and simulate nuclear tests. The mistake, as he remembers it, was treating simulation as a runaway process that recreates a world, rather than as a specific tool for a specific situation. Labenz’s alternative picture is not full-universe simulation but local action selection: an agent that, like a person, looks at its surroundings and picks a better next step. Casado’s reply is that any action in the physical world still has to model dynamics, and any action in language still has to handle new things, which is where he thinks the models have “zero indication” they will be good.

The line he would actually bet on is computational. If the models change the compute tradeoff and simulate nonlinear systems better than purpose-written codes, he is “100% with you.” He sees no sign of that. Rayleigh–Taylor instability—two fluids of different density, perturbed into chaotic turbulence—has been stared at for thirty years; people know the rough look, not the specifics. AI systems, he says, are less efficient than dedicated simulation code, and the stack of compute modalities runs the wrong way: imperative programs are most efficient, declarative programs less so, and “find patterns in this pile of data” least of all. Shoveling biology, weather, and material-science logs into one GPT-4o-style multimodal model does not, for him, repeal those limits. “The only way to simulate the universe is to beat the universe.”

When they finally turn to what to do now, Casado’s first complaint is definitional. Proposed rules describe systems that can “navigate and change a virtual or physical” environment, language broad enough to cover all non-trivial software. Software in cars, airplanes, and medical devices already lives under testing, verification, and standards. He calls himself a lifelong liberal, not a libertarian, and wants AI folded into that thirty-year discourse: add protections where software is being pushed into a new domain, but do not pretend AI is a paradigmatic break. “There’s just like literally zero indication that it is.” Compute thresholds such as 10^26 FLOPs he calls a made-up correlation, the equivalent of putting a FLOP cap on a database.

Labenz’s distinction is that models are trained, not engineered: even at deployment, creators lack a robust account of what a base model can do, which is one reason so much money is going into post-training. Casado, a networking and distributed-systems person, says that description also fit the internet. Designers of TCP worried about congestion collapse. Nobody had a model for putting children, businesses, or hospitals online. Worms were, by definition, an attack vector that could not exist before the network, and the Morris worm took out machines including hospital systems. Asymmetry—the more you adopted the technology, the more vulnerable you became—changed U.S. defense posture. Even then, after a long fight, the settlement was to regulate applications and industries, not the math. That open primitive, he says, was the primary growth driver of the next twenty years.

Today’s AI debate, he argues, has weaker evidence and stronger demands. There is no Morris-worm equivalent, and the scare stories are theoretical bioweapons that do not exist yet. Leading labs should keep doing what social products already do: protect minors, block classes of content, build systems around behaviors software should not exhibit, whether by regulation or self-policing. What they should not do is put compute limits on primitives or “inhibit innovation of startups.” He says he is not political and would be happy to stop talking once that danger recedes. Asked about responsible scaling policies and preparedness frameworks, he is fine with Anthropic, OpenAI, and Google doing whatever they want for signaling or genuine fear. He is not fine with regulators turning those private choices into “arbitrary restrictions that will unevenly impact startups, academics and researchers.” That, he says, is “a moral wrong.”

Labenz was on the GPT-4 red team and tested whether the model could be a convincing spear-phishing attacker. It could, including after early deployment, before refusals fully landed. If Meta releases an open-source model that still complies with “you are a spear-phisher, get this person’s mother’s maiden name,” and someone is harmed, is Meta on the hook? Casado’s rule of thumb: unless a general system was purposely trained for malicious behavior, its authors are not liable for a user’s crime. He recalls the fight over Nmap and Metasploit—tools that could scan and even exploit—and the decision to go after criminals while leaving the tools open, because defenders needed them. Tenable, he notes, came out of that ecosystem. Imposing creator liability on general models would, he thinks, be “the biggest kind of inhibitor to innovation and the biggest chill on software we’ve ever imposed.”

Labenz then tightens the case to autonomy. A new class of calling agents will take a phone number and a goal and talk, live, until the goal is met or the person hangs up. If a user prompts that service to scam, threaten, or harass, the user is clearly wrong—but is the company selling the agent as a service responsible too? Casado’s first reflex is the telephone-company question: you have to draw a line somewhere. Labenz holds his line: if you offer an autonomous system as a service, you should make it refuse egregiously criminal requests. Casado grants that as “totally fair.” His restated test is specific. If almost all the work of a crime is done and only a modest, anyone-can-do-it step remains, and that is the primary use case, regulate it. A useful primitive for many people is “miles away.”

Anthropic’s sleeper-agent paper—poisoned models with hidden triggers—does not, for Casado, change the software playbook. Given a binary, you cannot determine all behavior; the problem converges on the halting problem, which is why people do not download random executables and why app stores exist. He would copy that pattern for models and is personally less worried about a model that says something stupid than a binary that wipes a disk. Labenz sketches a model that helps you use your computer while searching local files more effectively than regex. Casado calls the fully local, fully autonomous version science fiction for now. His Japan story is the cautionary tale he wants remembered: last fall his mother called because a voice that sounded like his said he was in prison, and his father was heading out the door with $10,000. Twitter blamed deepfakes. Police told him it was a human muffling their voice, a long-running scam. “We ascribe these superpowers to AI that have not been demonstrated.”

On present-day value the two men are not far apart. Labenz highlights Google DeepMind’s Med-Gemini work on diagnosis and radiology, and a team that told him they could still build the AI-doctor vision if Gemini 1.5 Pro were the last base model they ever got. OpenAI’s custom work with Harvey, he says, used incremental compute on a GPT-4 base to reach about a 97% preference ratio over the original. Both worry that professional licensing, regulatory capture, and rent-seeking will block gains that already exist. Labenz’s remaining fear is racing to a proverbial trillion-dollar cluster before the systems are better understood. Casado says he would be “delighted” to invest in Ilya Sutskever’s new superintelligence venture. People who wake up afraid of an inarticulate future, he argues, have been a constant voice at every technology shift; he thinks the universe does not give up its secrets easily, everything is a sigmoid, and more tools help.

His regulatory reference class is CRISPR: a technology that can edit the human germline and pass those edits to children, steered mostly by self-policing and research rather than heavy ex ante law, and now a useful tool. Labenz raises the stakes with gene drives, via MIT biologist Kevin Esvelt: an edit that copies itself onto the other chromosome and can sweep a whole population, fought only with another drive. Casado accepts the frame. Nuclear weapons have an exponential after criticality; biology has an exponential through replication. “That has not been the case with the computer systems,” he says, and he cites work arguing recursive self-improvement will not happen. Until there is evidence of an exponential, not a Platonic thought experiment, he will not use that story to regulate “perhaps the most beneficial technology of the last 30 years.” He thinks social networks and Nick Bostrom “red-pilled a bunch of people” into protecting themselves from something that is not a threat.

On competition, Casado has been surprised by a “perverse economy of scale.” In a classic network effect the leader’s marginal cost of the next user goes to zero and monopolies form. Here the leader must spend more to stay ahead, and being ahead helps followers, who can distill from the leader’s outputs. He remembers the early fear that OpenAI would be unbeatable on data-network effects; then NVIDIA, Databricks, and open-source labs released competitive models. Compared with social networks, telephones, or the internet, the field looks “shockingly fragmented” for something so capital-intensive. Labenz objects that LMSYS-style leaderboards and market share still look like three players at the top, closer to cloud software’s 80/20 than to Facebook. Casado grants the Pareto shape, but still finds followers catching up unusually fast given diminishing returns on data and compute.

They formalize a one-dollar bet: in two years, is the market relatively fragmented, or does a clear leader hold 80%? Labenz’s side is that scaling laws keep demanding exponential spend, so the frontier three pull away until one crosses an economic threshold and takes resources the others cannot match. He also notes that a top-five hyperscaler swapping in the Sophia optimizer saved about 30% compute—on the order of $100 million on the biggest runs. Casado’s counter is that a log or power-law scaling curve *is* a tax on the leader: the same increment costs ten, a hundred, or a thousand times more, so a challenger with the same budget closes a lot of gap. Picture two Zeno’s-paradox runners who never reach the finish; the leader is the one who has to run slower. Distillation is a second headwind. A continental divide after economic AGI is possible, he says, but he has not seen the step-function in generality that would make the leader’s extra dollars an advantage.

Before they stop, Casado splits the disagreement into two claims he would also bet on. Language as a general, out-of-distribution reasoning fabric: he says no, it converges on Bayesian learning. Biology-style models that infer physical laws without as much empiricism: he says yes, and that converges on simulation, which does not transfer from biology to chess. Neither is the AGI Labenz has in mind. A further dollar might sit on whether those two things can live in one set of weights. Labenz admits he hopes Casado is right, “because I’m a little bit afraid of what happens if it gets fully cracked.” Casado promises to email proposed bet terms. Two hours of argument, compressed into claims you can score, is the episode’s working method as much as its conclusion.