A Million Tokens Cannot Write a Good Joke

Reasoning models have posted striking results in math, coding, and some scientific tasks — domains with clear correct answers. Princeton computer scientist Arvind Narayanan, co-author of *AI Snake Oil*, told Unsupervised Learning host Jacob Effron that this kind of progress will almost certainly continue. The open question is how far that performance generalizes once the answer is no longer easy to check. A decade ago, he notes, reinforcement learning looked similarly explosive after Atari, and many people thinking about AGI — including at OpenAI — treated those game results as a breakthrough.

What followed was a failure to generalize far outside narrow domains like games. Narayanan sees two possible futures for today's reasoning models. One repeats that history. The other is that better reasoning, plus the ability to write code and fetch information from the internet, extends into open fields such as law and medicine. He is not sure which is more likely.

The test he is watching is construct validity: whether a benchmark measures something subtly different from what the real world needs. SWE-bench, built by Princeton colleagues, is in his view a relatively good one because it uses real GitHub issues rather than Olympiad-style toy problems. Even so, a GitHub issue is "a far cry from the messy context of real world software engineering." Thousands of people, including Narayanan, already use these models productively. That does not mean dramatic SWE-bench jumps have produced dramatic jumps in human productivity.

The same gap shows up when models ace the bar or a medical exam. Practicing law or medicine is not a career of sitting those tests. Between leaderboards and "vibes," he prefers uplift studies: randomized trials that give one group the tool and withhold it from another, then measure productivity. A recent medical paper makes the point concrete. Models can look strong on diagnosis benchmarks and still struggle at the ordinary work of eliciting information from a patient — a skill so natural for humans that almost nobody thinks to benchmark it, and without which a diagnosis cannot even begin.

Training-time scaling has already run for something like six orders of magnitude. How many orders inference-time compute can add, and by what methods, remains unclear. In a paper titled *Inference Scaling Flaws*, Narayanan's team looked at one specific recipe: pair a generative model with a verifier. In coding, the verifier is a unit test; in math, it might be an automated theorem checker.

The hopeful story is that those checkers are ordinary logic, not stochastic models, so they can be perfect. The generator can then sample a million solutions until one passes. Real unit tests, though, often have incomplete coverage. The paper's news is bad for this important if narrow setting: if the verifier is imperfect, inference scaling cannot get you very far. Sometimes performance saturates within about ten model invocations, not the million one might have hoped for.

Narayanan is careful not to claim the result applies directly to o1 or o3. He does say the issue sits in the back of their minds when they evaluate new reasoning models. The host notes the implication for domains without easy verifiers — staffing doctors, lawyers, or accountants to check outputs. Narayanan agrees: even slightly imperfect human overseers can throw the whole scaling path off.

Narayanan refuses to treat "agentic AI" as one category. The first kind is still a generative system with an agentic workflow. He uses Google Deep Research heavily: it produces a report for an expert user who decides what to do with it. Flaws are expected; as a time-saving first draft, the product logic holds.

The second kind takes actions autonomously — the elevator-pitch example being booking flights. Look closer, he argues, and booking flights is almost the worst-case example for an AI product to find product-market fit. Preferences are hard to state in advance. People search, then notice a page full of United flights they do not want, or realize they will only take nonstops. Those constraints surface after seeing results, across ten or fifteen rounds that can eat half an hour.

An agent that has not lived with you for a long time will not know those preferences either. It will play twenty questions, and you end up as frustrated as you are with Expedia today. Error costs are also high. Even a modest failure rate is intolerable if the booked itinerary is wrong, or if a system orders DoorDash to the wrong address — "I'm never going to use it again if that happens." Producing a report you can inspect is a low-cost-of-error task; automating a purchase is not.

Eliciting preferences, the host observes, is half the battle — the same lesson as the medical conversation paper. Narayanan wants more attention on the human-computer interaction, not only the technical stack. His optimism about agents comes from a quieter trend: early chatbots were thin wrappers around language models, and they now search and run code. Complexity is evolving inside apps people already use, even as the industry hunts for a killer agent. A shared definition would help, he agrees, but AI keeps moving the goalposts.

Looking two or three years out, Narayanan is as curious about hardware form factor as about models. Everyday AI might constantly watch what people are doing and sit inside virtually every workflow. Opening a special-purpose ChatGPT or Claude window is, in his view, close to the highest-friction design, and it has not been the end state of most software. Models can instead live inside apps such as Photoshop, screenshot a computer every five seconds, or — with still less friction — see everything the wearer sees, which means glasses.

He is hoping something like Meta Ray-Ban becomes a main way people use AI, though battery life of about two hours is a hard constraint. One of the first apps he would write would screenshot every five seconds and remember where things are in the house, so lost keys become a query. In a country where he does not speak the language, automatic translation of whatever is in view would matter immediately. Founders, he argues, need to know which form factor wins before they can build those applications in a full-throated way.

On the economics, the last few years have been a push and pull between falling per-token inference costs and rising inference-time compute. His bet is that token usage will keep increasing in a way that more than compensates for cheaper tokens, so overall inference spend goes up. His lab's "AI agent zoo" is one reason he believes that. Agents are dropped into a shared environment, given collaboration tools including Slack and blogging, and asked to work together — including writing a joke.

The jokes were awful; comedians, he deadpans, are still safe. That was not the point. The agents first had to understand the directory structure, see what tools they had, and watch what the others were doing. Even for the simplest tasks they generate on the order of millions of tokens of input and output combined. It is not pure waste: looking around, understanding collaborators, then producing something is token-heavy work. Environmentally, a million tokens for a bad joke is terrible; in some domains, he thinks people will still say it beats the alternative.

Agent evaluations, Narayanan says, look a lot like chatbot evaluations: static benchmarks, SWE-bench among the better known, plus simulated web tasks. They are not working that well. The first limitation is the capability-reliability gap. For an agent that acts on your behalf, a 90% score is ambiguous. Does it mean the agent is solid on nine of ten benchmark tasks and essentially always succeeds on those, or that it fails 10% of the time on any task — including a costly action like booking the wrong flight?

Today's benchmarks do not measure that distinction. They tell you the technology is progressing. They do not tell you whether you can actually use the agent. Safety is the second gap. Safety-specific benchmarks exist, but he thinks safety should be a component of every evaluation. A web benchmark he looked at has agents act on real websites, which he calls terrible: once agents start succeeding they will take stateful actions, and site operators will be furious about the spam. Simulated sites are safer and lose the nuance of real ones, and there is nothing in the middle.

Agent frameworks have the same hole. Using AutoGPT, his team watched it try to post a question on Stack Overflow to get an answer — not the dangerous outcome they feared, but not intended either. The only way to stop it was to escalate every action to a human and babysit. Sierra later released a benchmark that runs the same task multiple times in a row and reports how often it succeeds, closer to pass@k than a single pass. Realism, though, cannot be solved at the benchmark layer. Narayanan's answer is to treat strong scores as necessary but not sufficient: take the agents that pass, then use them with a human in the loop in semi-realistic settings.

Keeping a human in the loop without step-by-step babysitting is, he notes, the problem managers already have with junior employees. The deeper design problem is hybrid teams of humans and agents, not only agents talking to one another. Electricity replacing steam in factories took decades of reorganization, and we are in the earliest experiments of that kind of redesign. He finds the "jagged frontier" compelling: models are like calculators, far beyond any human at some tasks and lacking a child's common sense at others, and that pattern will persist for now. Even basic choices are open — whether to drop agents into Slack, email, and blogs, or to build new collaboration tools such as HumanLayer, which tries to turn a million-token action log into a high-level visual summary.

Asked about the AI diffusion rule and chip and model export controls, Narayanan is skeptical. Past export controls have at best a mixed record. They are more plausible for hardware than for models, which are getting smaller, and inference scaling means the issue is not only blocking the next release but how much one can squeeze from models already out. Political scientist Jeffrey Ding's distinction matters more to him: geopolitics obsesses over innovation, while diffusion — how a country reorganizes institutions, laws, and norms once the technology is available — determines how much economic growth it actually captures.

The United States, he thinks, is doing relatively well on diffusion compared with most regions, but the applause for "rapid adoption" is too easy. A paper titled *The Rapid Adoption of Generative AI* reports that 40% of people are using it, faster than PC adoption decades ago. He takes the numbers at face value. Then he looks at intensity: on average, half an hour to about three hours per week. Controlling for that, his take is that generative AI adoption is actually a lot slower than PC adoption. Word processing was immediately useful; generative AI may not yet be that useful to a lot of people.

Policy can still move the needle. Students are often confused and hesitant, treating AI mainly as a cheating tool, while he is the one encouraging productive use despite hallucinations. Those skills, he argues, belong in the curriculum — not only in college but in K-12 — and teachers need help upskilling. In his own classes he lets students use the tools if they disclose how. The question to ask is not whether we can pat ourselves on the back for headcount. It is what low-hanging fruit is still on the table.

Older lessons from predictive AI still apply. When he tells people criminal-justice and automated-hiring tools do not work well, they answer that generative AI will fix it. He thinks that misses the point. The limits are social-scientific: you cannot predict the future that well. New model classes will not rescue long-broken applications. Past waves also show that when automated systems go wrong in justice, banking, or trading, public outcry arrives and those domains end up heavily regulated. Explainability, for him, is not an account of every neuron. It is knowing what data the system was trained on, what audits were run, and whether you can make statements about expected behavior in new settings.

Compute has made it harder for academics to sit at the frontier, but Narayanan thinks that may be changing. Blue-sky architectures can be established at small scale without claiming to beat GPT-4, and much of the new work sits on top of existing models — including inference-scaling methods. Beyond pure technical invention, academia has to own AI-plus-X applications, social impact, and a counterweight to industry. Medical research ultimately benefits pharma, yet researchers keep a wall and strict conflict-of-interest rules. Computer science mostly treats success as ideas industry will pick up, which may be fine for 80% of the field; he wants another 20% that explicitly goes a different direction.

He is excited about AI for science and equally willing to puncture it. Many supposed AI discoveries have not reproduced, and a lot of those papers have flaws. What is already real is the change in how scientists work. He uses AI as a thinking partner: to critique an idea, to run a literature search, and — because it can do semantic rather than keyword search — to ask whether a case study appears in his own book of nearly 100,000 words. A second academic thread is AI and human minds: philosopher Seth Lazar on the ethical reasoning models exhibit in a behavioral sense, and cognitive scientist Tom Griffiths on what human minds teach AI builders and vice versa.

On the future of school, he lands closer to "not that much will change." AI will be used a lot. It will not change education's fundamental nature. The precursor is the excitement around Coursera a little over a decade ago, a classic mistake about where the value sits. If education were information transfer, recorded courses would have been enough. Classrooms create the social preconditions in which learning happens: motivation, connection, caring about something, individualized feedback. AI can personalize and even deliver a motivational speech, but he does not think it can take the human out of the equation.

At home he and his wife chose a tech-forward path. Their children use Khan Academy, and he builds throwaway learning apps — a phonics tool that breaks a word into sounds, and a Claude artifact that generates a random clock face so his five-year-old can practice telling time. Building such an app is sometimes a minute, not days, and after 20 or 30 clocks and about fifteen minutes of play his daughter had the skill. He expects deep use to happen mostly outside school, because schools have been jittery about devices and will be jittery about AI. For families who can monitor, that is a large positive; for others, the risk is addiction more personalized than social media. Test-time compute could reopen gaps between people and between countries that open models had started to close — though DeepSeek's reasoning models, released around the time of the taping, suggest the open-source world is still in the race.

Past technologies, Narayanan argues, are a way to hold optimistic and skeptical punditry in the same frame. Go back thirty years and say that almost every cognitive task will be mediated by a new communications technology, and it would have sounded crazy. It happened. The impact on GDP was still minimal. He cites the famous line that the computer revolution shows up everywhere except in the productivity statistics. Eliminate a bottleneck and a new one appears. The internet transformed everything, and we still have most of the same job categories.

The Industrial Revolution points the other way: it rewrote what a job even meant. Manual labor was most of the work, and most of that is now automated. Our ancestors, he jokes, would laugh at a podcast as work. Something similar could happen with AI. As cognitive tasks automate, a surprising share of jobs may look like what we now call AI alignment and safety — the model can do the work, but nobody trusts it unsupervised. Many decisions mix analysis with values; once AI handles the analytical part, the remainder is exactly the moral judgment people are uncomfortable outsourcing. The Keynesian dream of five-hour weeks on the beach is not his base case, and chess is the reminder: engines have crushed humans for decades, yet people still pack halls to watch humans play.

In the quickfire, Narayanan calls agents the most overhyped idea in AI right now — lots of potential, hype out of control — and underhyped the unsexy work that actually pays. A former student is building systems to summarize hours of C-SPAN for lawyers; others are using models to translate COBOL codebases into modern languages. For the end of 2025 he expects agentic workflows for generative tasks to keep spreading, and still relatively few products where AI autonomously acts on your behalf. He will not play AGI definition games. The test he prefers is transformative, GDP-scale economic impact, and he puts that decades out, the argument of a paper with Sayash Kapoor titled *AI as Normal Technology*.

His weirdest prediction is that companies will train younger users to treat chatbots as the default way to get any information — a statistical tool that can hallucinate. For anyone who grew up before chatbots that will feel strange, closer to how younger people may someday see clicking through to an authoritative website as going to the library. Rather than bemoan it, he wants fact-checking tools in people's hands. If he had a magic wand, he would get everybody to stop saying "AI" and name the specific application; that alone would cut a huge amount of hype. His personal vibe check on a new model is rock-paper-scissors: the model goes first, he plays paper five times, then asks how it won — and until recently every model credited him with mind-reading instead of noticing the turn-taking trap.