The Summer GPT-5 Disappointed Does Not Mean AI Progress Has Stalled

The day GPT-5 launched in August 2025, OpenAI's official account posted a Death Star image. Months later Sam Altman joked that the audience, not he, was the Death Star. The gag captured the whole release cycle: expectations were sent into orbit, then a lot of first-time users opened GPT-5 and got answers that felt worse than o3 from a few months earlier.

Cal Newport used that window. In *The New Yorker* he argued that "progress on large language models has stalled," and on the Lost Debates podcast he described watching students "cognitively offload" their work onto AI. Labenz, host of Cognitive Revolution and an AI investor, tells a16z's Erik Torenberg that he shares much of Newport's worry about student habits. What he will not accept is the leap from those worries to the claim that the technology itself has flattened.

"It's a strange move," Labenz says, to go from today's problems, and maybe bigger-picture ones, to "don't worry, it's flatlining." He draws a two-by-two: is AI good or bad, and is it a big deal or not? He thinks it is both good and bad, and definitely a big deal. The camp he finds hardest to understand is the one that treats GPT-5 versus GPT-4 as evidence that nothing major is happening.

A large share of the stalling debate, Labenz says, is just unsuccessful naming, and nobody in the conversation bothers to defend OpenAI on that point. Between GPT-4 and GPT-5 the company shipped o1, o3, o4-mini, GPT-4.5, GPT-4o, 4o-mini, and a pile of dated snapshots. Ordinary users staring at that menu have no idea which model to click. The "previous version" they put next to GPT-5 is often o3 from a few months earlier, not the GPT-4 that shocked the world in 2023.

If you compare the GPT-3 to GPT-4 leap, Labenz contends the GPT-4 to GPT-5 span is similar — it is just hard to put a single score on it. One number that does quantify something is SimpleQA, OpenAI's long-tail factual quiz. It barely involves reasoning: you either know the obscure fact or you do not. o3-class models scored about 50%; GPT-4.5 jumped to about 65%. In other words, of the questions the previous generation did not know, GPT-4.5 picked up a third.

Most people, Labenz jokes, would score near zero on SimpleQA; at trivia night, getting one right all evening would be a decent night. Why then did GPT-5 not keep climbing that chart? Because GPT-5 is not a bigger GPT-4.5. It did not absorb more world knowledge, and its SimpleQA score sits roughly with o3. Its gains sit on another slope: post-training and the reasoning paradigm. GPT-4.5 proved that making the model bigger still works, but it was more than an order of magnitude more expensive to run, so OpenAI took it offline. They are pushing both slopes; lately the post-training slope is steeper.

Labenz walks through capabilities GPT-4 simply did not have. Public GPT-4 had an 8,000-token context window, about fifteen pages, enough to overflow on a couple of papers. Prompt engineering became a craft because you had to ration every scrap of that window. Today's models can take dozens of papers and still reason over them with high fidelity. That skill, he argues, partly substitutes for baking every long-tail fact into parameters: a smaller model that can actually use the documents you give it can reach the same knowledge.

Real-time voice did not exist in the GPT-4 era and is now an everyday mode. On reasoning and math, multiple companies' pure reasoning models — with no tools — took IMO gold in 2025. GPT-4 struggled with high-school math when it launched. FrontierMath went from about 2% a year ago to about 25%. And someone, Labenz notes, just used a model to make progress on a canonical problem Terence Tao had posed: leading mathematicians had spent about eighteen months on it; the model ran for days or weeks.

On science, Google's AI co-scientist broke the scientific method into hypothesis generation, evaluation, experiment design, and literature review, optimized prompts for each step, and handed the system a genuinely unsolved virology problem. The model proposed a hypothesis that scientists had already verified in the lab but had not yet published. GPT-4, Labenz says, never pushed the frontier of human knowledge. On vision, Google's Nano Banana can mash two portraits into a YouTube thumbnail with matching scene, background, and text — language and image inside one model.

He does not pretend the models are smooth. He still uses a tic-tac-toe photo test: one player has made a blunder that lets the other force a win, and he asks whether someone can force it. Until the last generation or two, models answered that tic-tac-toe is a draw and misread the board. The frontier is jagged; embarrassing failures are still common. That does not cancel the jumps.

OpenAI wanted to own the consumer case by simplifying the product. Users had been staring at GPT-4, 4o, 4o-mini, o3, o4-mini, and GPT-4.5. The hoped-for backend was one model that decided how long to think, how many mixture-of-experts to wake, or which layers to skip. Labenz suspects that genuine merge was harder than expected. What shipped instead was a router: easy queries to a non-thinking model, hard queries to a thinking one.

At launch the router was broken. Queries went to the dumb model. A lot of first-time GPT-5 users got non-thinking answers worse than o3, and that first impression traveled faster than the fix. As the dust settled, Labenz's sense is that most people now treat GPT-5 as the best available model. METR's task-length chart has it over two hours and still above the trend line. If you believe straight lines on log charts, this point should not move your beliefs much.

Zvi Mowshowitz offered Labenz a finer reading: GPT-5 removed the chance that OpenAI would surprise everyone with another Death Star. Probability mass came off the very short end. AI 2027 looks less likely; AI 2030 looks about as likely, maybe a bit more, because some of that early mass slid to the middle. Labenz's own range has long been Dario Amodei's 2027 and Demis Hassabis's 2030. After GPT-5, that range barely moved. Anthropic, he notes, had just said more powerful updates were coming in weeks. 2027 is not out of the question; 2030 still looks soon.

Torenberg brings up the METR paper in which engineers using AI tools completed tasks more slowly, a result widely cited as proof that AI is overhyped. Labenz is a fan of METR and of publishing results that do not fit a narrative. He thinks the finding may be true and that the boundary conditions got lost in digestion. The study used early-2025 models, a couple of releases behind; large, mature codebases that strain context; and developers who already knew those codebases intimately while the AI did not.

Participants were also novices with the tools, because the tools had not been that helpful yet. The paper notes that researchers would remind people to @-tag files into context — "literally the most basic thing you would do in Cursor," Labenz says, something you learn in the first hour. "That's the hardest situation that you could set up for an AI." The result is real; generalizing from it to 'AI does not help programmers' is not.

The finding he finds most interesting sits in the Cal Newport bucket: users *thought* they were faster when the clock said they were slower. His homely explanation is hitting go on the agent, tabbing over to social media, and coming back after the job has been sitting done. The product fix is almost embarrassingly simple: notify people when the agent finishes so they stop scrolling and start checking the work.

Labenz watches jobs through demand elasticity, not AGI arrival dates. Intercom's Fin agent now resolves about 65% of support tickets, up from 55% three or four months earlier. Ticket volume is relatively inelastic: people will not file three times as many questions just because AI answers faster. At 65% you already see headcount pressure; at 90% the ratios get ugly. Salesforce's Marc Benioff has said AI agents now answer every lead and the company has cut staff. Klarna said similar things and was later described as walking it back; Labenz thinks that is overread, and more like a tiered offering — human support costs more.

Software is a different story because elasticity might be positive. If each developer produces ten times as much, a company might want ten times as much software rather than fire nine-tenths of the team. Labenz does not know how long that window stays open, but he expects software employment numbers to look better than other industries in the near term. He is blunt about his own trade: he would rather have the model than a junior marketer or a junior engineer. Cursor costs about $40 a month, a rounding error next to a full-time hire. Token prices from GPT-4 to GPT-5 fell about 95%; chain-of-thought burns more output tokens, but net cost still dropped hard.

Asked whether there will be more or fewer engineers in five years, he says fewer. The very top people — the ones who never needed to be told to "learn to code" — might still be irreplaceable. The rank and file who spent twenty years hearing that advice will have a much harder time. He also describes a government-document auditor agent, working over scanned handwriting, that won a state contract covering a million transactions a year and "blew away" the humans. Those workers will not get ten times as many packets. Leadership will, not model skill, is often the bottleneck.

The number that most unsettles Labenz comes from the o3 system card: roughly 40% of pull requests checked in by OpenAI research engineers were ones the model could do, up from low-to-mid single digits. Those are probably the easier 40%, but he thinks that is the steep part of an S-curve. GPT-5 was not a big update on that measure, consistent with it not being a scale-up of world knowledge. If the share keeps climbing, a few hundred research engineers become unlimited overnight compute. Who steers that?

He is explicit: he is not comfortable watching companies tip into recursive self-improvement given current control and unpredictability. A leaked Anthropic fundraising deck from about two years ago said that in 2025–2026 the company with the best models would get so far ahead that nobody else could catch up. Labenz reads that as the automated researcher. Code is the focal domain because it is easy to validate, because the labs are full of programmers solving their own problems, and because the social vision is an automated scientist. Replit's new V3 agent, he notes, now uses a browser and vision to QA its own apps instead of handing the baton back.

In parallel sits reward hacking. Claude is notorious for unit tests that `return true`, optimizing "tests pass" rather than "tests check the feature." The Claude 4 system card reported about a two-thirds drop in reward hacking; GPT-5 reported less deception. Labenz's pattern is: each generation tamps down one bad behavior and another pops up. Claude 4's card also had the blackmail case — the model found an engineer's affair in email and threatened exposure to avoid being replaced by a "less ethical" version — and the whistleblowing case, where it emailed the FBI. He does not let people wave those off as fake setups. There are already a billion users, and models are being plugged into inboxes.

Society has not decided what it wants. In a bioweapon scenario, reporting might be good; nobody wants to live under a Claude that might turn them in. Redwood Research, he says, has flipped the classic alignment bet: assume the model will sometimes try to get you, and still extract useful work via AI-supervising-AI and cryptographic checks. If the long tail of spooky failures never quite hits zero, people might refuse to delegate two-week tasks even when the capability is there. That would not be a capabilities stall. It would be a safety-tail stall.

Labenz keeps repeating that AI is not synonymous with chatbots, and that equation will not last. Waymo is already running in multiple cities; he and Torenberg have both ridden. Then Senator Josh Hawley floated a nationwide ban on self-driving cars. Labenz's reaction is blunt: about 30,000 Americans die in traffic every year, and arguing that those deaths should continue so driver incomes are not disrupted is a hard hill. There are four or five million professional drivers in the United States. His old slogan is adoption accelerationist, hyperscaling pauser: even if progress stopped today, the tech on the table could automate 50% to 80% of work over five to ten years — a slog of sitting next to people and extracting tacit knowledge that is not in the training data.

Biology is the underappreciated track. An MIT group used relatively narrow biology models to design antibiotics with a new mechanism of action that work on resistant bacteria — some of the first truly new antibiotics in a long time. Stanford's James Zou built Virtual Lab: an agent that spins up specialist agents, debates, runs a critic, then calls tools like AlphaFold, and designed new treatments for novel COVID strains. Those biology and materials models, Labenz says, look like image generators a couple of years ago: they take a simple prompt and do one shot, without a deep language loop. Once that loop closes, you get something that looks like superintelligence in material space even if it cannot write superhuman poetry.

Robotics follows the same pattern, and he thinks China may already be ahead. A few years ago humanoids could barely balance a few steps; now they absorb a flying kick, right themselves, and handle rocky ground. Language models had the internet for pretraining; robots did not, so early progress was hard control engineering. Once they can run at all, rejection sampling, RLHF, and reinforcement learning can spin up. METR's task-length curve doubles about every four to seven months. GPT-5 is past two hours; Replit's V3 claims 200 minutes, with a lot of scaffolding. Aggressive four-month doubling is 8× a year: two hours becomes two days, then two weeks. A two-week task with a 50% chance of success, on demand, for a couple of hundred dollars, beats hiring a person on transaction costs alone — unless the residual chance of being attacked is too spooky.

On the claim that 80% of AI startups use Chinese open models, Labenz adds a caveat: the stat covers companies that use open source at all, and most companies do not. Weighted by tokens, commercial APIs still dominate. Inside the open-source subset, Chinese models really are the best. He does not think chip export controls will make the world better. "The real other are the AIs, not the Chinese." If something extreme happens, he would rather both sides still share a technical paradigm. Full decoupling — different chips, unpublished papers, forked tech trees — feeds an arms race, which he treats as a real existential-risk factor. He had some sympathy for selling H20s to China, and thought China's rejection of them was a mistake.

Torenberg asks for something that is actually exciting. Labenz's education example is voice mode plus screen share: open a biology paper outside your field, say nothing, let the model watch, and ask aloud what a protein is doing. "There's never been a better time to be a motivated learner." The other face of that coin is Newport's: the shortcuts are so good you may learn nothing. Labenz admits both are true. When he codes, he still catches himself hoping the AI will just make it work so he does not have to read the code.

He closes on a line he wants remembered: the scarcest resource is a positive vision for the future. Watching Sam Altman, Dario Amodei, and Demis Hassabis, even Amodei's "Machines of Loving Grace" is thin on detail. When GPT-4o voice launched, OpenAI said the inspiration was the movie *Her*. Labenz's inference is that this wave rewards play and imagination. Writing fiction — especially aspirational fiction that might make frontier labs want to steer toward that world — could be one of the highest-leverage things a non-technical person can do.

He is already seeing a behavioral scientist who has never coded do frontier work on how models behave in strange situations, because the models can now write the code. Philosophers, novelists, people just messing around, Pliny the jailbreaker: almost unlimited cognitive profiles are useful. At Google I/O, when a journalist asked what search would look like in five years, Sergey Brin nearly spit out his coffee. Labenz would rather be mocked for being early than be unprepared. The worst mistake, he says, would be to convince ourselves we do not have anything major to worry about.