Arvind Narayanan on AI Scaling, Real-World Risks and the Limits of Model Benchmarks

Arvind Narayanan believes training data may limit the gains from building dramatically larger language models. More compute can still help, he says, but he expects fewer successive jumps in parameter count and capability of the magnitude seen in earlier model generations.

Harry Stebbings speaks with Narayanan, a Princeton computer science professor whose work includes technical AI research, the social consequences of AI and advice to policymakers. He is also a co-author of the book AI Snake Oil.

Narayanan spent years studying cryptocurrency because he hoped decentralization could improve society. By 2018 he had become disillusioned: in his view technology was not the main obstacle to serving unbanked people, and replacing flawed institutions with software was not a substitute for the difficult work of reform.

He regards AI as a net positive for society despite its harms, unlike Bitcoin in his assessment. He will not predict whether AI investment is a bubble, but thinks generative AI companies have made serious product mistakes.

ChatGPT revealed many applications its creators may not have anticipated. Narayanan says some developers drew the wrong lesson, assuming that releasing a powerful model was enough and users would discover the rest. He argues that product-market fit and building things people want still matter.

Asked whether more compute yields more performance, he says past gains came partly from bigger models and more training data. He regards the step from GPT-3.5 to GPT-4 as largely a size increase, while noting that the relevant training details had not been made public.

His prediction is that there will be few, perhaps no, further model cycles with almost an order-of-magnitude jump in parameters and corresponding gains. Companies have already used much of the data they can obtain, he says. Compute can still improve models, but perhaps less dramatically than before.

Additional compute can also help produce a smaller model at a similar capability level. Narayanan points to models becoming smaller and cheaper than the initial GPT-4 generation while remaining capable, and is skeptical that a future GPT-5 will produce as large a leap as earlier generations.

Stebbings asks about unused video and synthetic data. Narayanan argues that even a very large quantity of video may yield comparatively few text tokens once speech is extracted and duplicated material removed. Training directly on video could unlock multimodal capabilities, but he doubts it will fundamentally transform text capabilities in the same way.

He recalls surprise that a model trained predominantly on English could acquire some competence in other languages from relatively little multilingual text. His interpretation is that many such capabilities are already expressed in available training material, limiting the likelihood of comparable surprises merely from adding more similar text.

Narayanan distinguishes synthetic data used to improve quality from simply multiplying data volume. Generated examples can supplement underrepresented languages or provide worked mathematical problems. He is skeptical that successive generations of synthetic text can supply unlimited new information, arguing that quality matters as much as quantity.

Stebbings observes that much organizational work is tacit: a whiteboard discussion may never appear in the final document. Narayanan agrees that passive ingestion of web data cannot capture every skill. He compares some organizational knowledge to riding a bicycle and says models may need to learn through use and feedback inside organizations.

That learning cannot always come from passive observation. Narayanan expects deployment and feedback to be slow, comparing it to the gradual rollout of self-driving vehicles as operators collect experience and improve reliability. His analogy is an expectation about adoption, not evidence of an identical safety profile.

He says capability is not always the chief barrier to adoption. If models were deployed everywhere they can already perform useful tasks, he believes the economic effects would be striking; cost is among the practical constraints pushing developers toward smaller models.

Small models can also run on a device, potentially addressing privacy concerns for assistants that handle phone conversations or desktop activity. Local operation can reduce the need for dedicated inference servers, although smaller size does not by itself settle every privacy question.

Stebbings asks whether falling hardware costs will eliminate the problem. Narayanan invokes Jevons' paradox: cheaper use can expand demand. He predicts total inference spending may rise as models are used in more applications, even while a given operation becomes cheaper.

Simple text chat may become inexpensive, but always-on analysis of email and large documents can consume substantial resources. For coding, he says repeated attempts and selection of a better result can use additional computation to improve quality, creating demand for more inference even as unit costs fall.

Training compute builds a model; inference compute runs it for users. Narayanan says inference can dominate lifetime costs for widely used models. Training a smaller model longer can raise the initial expense while reducing the cost of serving the same workload, potentially lowering total cost.

Stebbings cites the claim that frontier models will require a new data center for every training run as hardware and models advance. Narayanan says both are progressing rapidly but expects exponential-looking growth eventually to slow. He cannot say when, and sees eventual model commoditization as possible.

Evaluating models is difficult, he says, because strong benchmark scores can coexist with disappointing real-world use. Developers may intentionally or unintentionally optimize for the test while missing mistakes that users encounter in practice.

The bar exam and medical licensing examples illustrate the gap: answering examination questions is not the same as performing a lawyer's or doctor's work. Benchmark contamination is another concern if a model has encountered answers during training. Narayanan prefers evidence from professionals using systems in their actual work.

For discussing AGI, Narayanan uses a practical definition: AI capable of automating most economically valuable tasks. This definition does not require settling whether a system is conscious or thinks like a person, and reaching it would nonetheless profoundly change society.

He cautions against confident CEO timelines, noting that predictions of imminent AI have recurred for more than half a century. He compares progress to climbing a mountain: each apparent final step can reveal more complexity. In his view the limits of simple model scaling are one such emerging obstacle.

Stebbings asks whether a company can pursue AGI while building useful products. Narayanan says it can, but agrees that management needs discipline to prevent competing priorities from distracting one another. He interprets staff departures and differing company emphases as signs of that tension, rather than proof that the combination is impossible.

He believes some developers previously treated product development as unnecessary because they expected general intelligence to arrive quickly. He cites the delay before ChatGPT's mobile apps as an example of what he sees as a model-first philosophy. He thinks the industry's greater attention to products is a healthy change.

The cost of frontier training raises the question of whether small companies and universities can compete. Narayanan says continued dependence on ever-larger models is not inevitable: if models become commodities, useful ideas in layers above them, including agents, could come from small startups or academic labs.

Stebbings proposes a small group of foundation-model providers financed by large existing businesses, with other companies building on top. Narayanan considers that plausible and says competition and market concentration deserve regulatory scrutiny; he notes interest from US, UK and European authorities.

Narayanan argues that many issues labeled AI regulation are better understood as regulation of harmful activities. He cites fake product reviews: rather than asking a model to determine whether a review is genuine, authorities can target the deceptive review regardless of whether a person or AI wrote it.

He broadly agrees with allowing experimentation while watching for harm, but says some harms are already known and serious. Non-consensual intimate deepfakes have affected many people, primarily women, and he welcomes government action addressing them.

Stebbings raises the risk that deepfakes will cause people to disbelieve real news. Narayanan calls this the liar's dividend. He is more skeptical about AI-generated election misinformation as a distinct new danger, and the two disagree over how much more persuasive realistic fabricated media might become.

Stebbings says a credible media company could create a convincing fake of a political leader. Narayanan accepts the example but emphasizes the importance of source credibility, predicting that people may increasingly turn to outlets they already trust. That is a forecast, not evidence that audiences will all trust the same sources.

Stebbings remains concerned that AI images can spread widely and incite action, including by reinforcing existing beliefs. Narayanan argues that misinformation often confirms rather than creates beliefs, and says crude manipulated material could already cause harm before generative AI.

He acknowledges that AI makes image manipulation easier, but warns that treating misinformation primarily as a model problem may distract from social trust and distribution. In his view, social media platforms bear particular responsibility because reaching and persuading audiences is central to the problem. Stebbings remains more worried about the content itself.

Narayanan is particularly worried about non-consensual intimate deepfakes, which he says received more public attention after high-profile examples involving Taylor Swift. He stresses that such abuse existed for years before the latest generative AI wave.

He also points to costs shifted onto schools when students use AI for homework. Teachers may need to redesign instruction and assessment, and he doubts AI-generated text can reliably be detected. He does not conclude that AI necessarily worsens education, but argues companies should ideally bear some of the adaptation costs.

Stebbings questions claims that AI can put a general practitioner in everyone's pocket, pointing out that doctors physically examine patients. Narayanan agrees with the concern: technological shortcuts can look appealing when healthcare is expensive or access is difficult, but they do not automatically repair the underlying system.

He favors responsible integration into healthcare, where technology is already used in imaging and where AI might assist with diagnosis or summarizing notes. Whether a person who cannot quickly see a doctor should ask a chatbot about symptoms, even with safeguards, is a harder question to which he says he does not know the answer.

On education, Narayanan distinguishes highly self-motivated learners from the many people for whom social relationships are central to learning. He warns that AI developers may mistake their own experience for that of a typical student.

He describes himself as a self-directed learner who finds generative AI useful for asking questions and checking his understanding alongside books. Still, he is skeptical that these tools will soon become the default way most people learn.

He considers immediate fears of broad job replacement exaggerated. He cites bank tellers as an example of a technology changing the composition of work: ATMs made branches cheaper to operate, and banks expanded branches that still needed people for other tasks.

His broader point is that jobs consist of bundles of tasks, and automating one task does not automatically automate an entire job. He acknowledges that some occupations, including translation and stock photography, have already felt substantial effects; he does not claim that every worker is insulated.

Asked whether AI is more dangerous than nuclear weapons, Narayanan calls the comparison a category error: a nuclear weapon is itself a weapon, while AI is a general technology that can enable harmful uses, including attacks on critical infrastructure.

He opposes relying on closed models as the main safety strategy, arguing that capable models already run on personal devices and that universal international restrictions are improbable. Instead he urges society to plan for widespread access and strengthen defenses accordingly.

As an analogy, he points to software developers using automated tools to find and fix vulnerabilities before release. His hope is that AI will similarly help defenders more than attackers; he presents this as an aspiration, not a demonstrated guarantee.

Narayanan says the short interval between GPT-3.5 and GPT-4 initially exaggerated his impression of the pace of progress, because GPT-4 had already been in training for much longer. At the time of the interview, he does not see later models as having qualitatively surpassed GPT-4 in ordinary use, a subjective and time-specific assessment.

He thinks further advances may require new scientific ideas rather than simply making models larger. He also argues that science-fiction images distort fears about current AI consciousness and agency; he favors transparency, monitoring and regulation as systems develop. These are his views, not a settled scientific consensus.

In the quick-fire round he reiterates that leaderboards often fail to capture usefulness in real settings. If he could change OpenAI, he would prioritize public transparency about AI development over a company's commercial interests.

His preferred vision for agents is mundane but useful: a device that understands a nuanced instruction and books a flight or helps build an app. Asked about competition with Nvidia, he notes the irony that the hardware company is moving into services while other companies move toward chips.

Narayanan says policy work can be frustratingly slow, but rejects the idea that policymakers are simply unintelligent. He believes the worthwhile minority of outcomes makes technical expertise in policy important. He expresses optimism about the then-current US approach, while saying he has not closely examined every other country's policy.

He says he appreciates Yann LeCun's view that language models alone are not a direct route to superintelligence and that more breakthroughs are needed. His final concern is AI's influence on children: he urges technologists, parents and policymakers to consider outcomes that could be profoundly good, bad or somewhere in between.