How quickly could AI improve itself? Four forecasters disagree

Jane Street's Ron Minsky recalls hosting a 2011 debate between Robin Hanson and Eliezer Yudkowsky about whether AI could rapidly improve itself. Fifteen years later, he brings forecasters together for a discussion rather than another formal debate, asking what paths the technology might take and what they would mean.

Minsky introduces Jaime Sevilla of Epoch AI, Ege Erdil, Ryan Greenblatt and Daniel Kokotajlo. Their forecasts differ in speed, but he says they also share assumptions. He begins by asking for plausible scenarios rather than a single definitive prediction.

Erdil questions whether a median forecast captures a world with very different possible paths. If one assigns substantial probability both to rapid success and to a much slower trajectory, the numerical middle may describe neither scenario well.

One important uncertainty, Erdil says, is how much an AI can learn from tasks that can be simulated or verified inside a lab. If that training yields general planning and cooperation skills with little additional real-world experience, he thinks progress could accelerate quickly.

He assigns roughly a 20% chance to that sort of rapid development in the next few years. In the larger share of scenarios, he expects further progress to require physical inputs, including more computing power and real-world data, that a lab cannot scale as freely.

Erdil expects current compute growth to slow eventually and, with it, the pace of progress. His rough median for a world where nearly all jobs can be automated is around 15 to 20 years, while he retains substantial probability for a much faster scenario.

Minsky asks whether the constraint is the need to spread AI across the economy or simply to obtain more data and equipment. Erdil distinguishes the two: more chips and power might be necessary for a feedback loop even if sectors such as law or medicine adopt slowly and aggregate GDP does not immediately surge.

Kokotajlo offers a faster scenario, while noting uncertainty about the exact year. He puts roughly a 50% chance on full automation of AI research and development by the end of 2028, with labs continuing to improve models at research tasks as they compete.

In that scenario, AI systems conduct experiments, analyze results and coordinate research much like a company inside a company. Humans might still read reports and approve important decisions, while the day-to-day work is automated.

Kokotajlo expects that feedback to accelerate progress substantially, potentially producing systems better than the best humans on most tasks within months to a year. He then imagines rapid advances in robotics and physical production, followed by major changes in economic and military power.

His concern is that if highly capable systems are not robustly aligned with human intentions when they gain authority, humans could lose control. He calls this his single most likely scenario, while emphasizing that many other futures remain possible.

Sevilla describes Epoch AI's starting method: plot important measures of AI development and impact, then ask what happens if their trends continue. He treats this as a baseline against which to test arguments for acceleration or slowdown, rather than proof that an exponential trend lasts forever.

He places that baseline between Kokotajlo's rapid upward bend and Erdil's slowdown from input constraints. Recent rates already reflect some of the forces driving and limiting progress; Sevilla wants to identify the relevant measurements and watch whether they move faster or slower than expected.

Sevilla says competing forces can offset one another: easily automated tasks and readily available compute may run out, while AI-assisted research could accelerate development. He would first extrapolate the current trend locally and update as new data reveal which forces dominate.

Greenblatt asks which trend deserves extrapolation. Extending recent company revenue or automation growth for several years could produce extraordinary results, he notes; a qualitative comparison between GPT-3 and current systems also shows a substantial jump in what AI can do.

Minsky cautions that many apparently exponential curves eventually become S-curves. Population projections and the end of rapid processor-clock improvements are examples he raises. He asks how a forecaster decides when extrapolation remains useful.

Sevilla agrees that selecting a trend and estimating its limits require expertise. He says short-term extrapolation has nevertheless worked surprisingly well for some AI revenues, even as the eventual physical and economic limits are undeniable.

The group considers evidence that some capability measures accelerated around the introduction of reasoning-focused systems. The key question for Sevilla is whether that bend represents sustained acceleration or a one-time shift before another roughly exponential period.

Minsky distinguishes exponential growth, where a quantity rises by a roughly constant percentage, from a rate that itself increases. The mathematical singularity is not literal infinity in a physical world, he says; the concern is that the change could become extremely rapid.

Erdil says some of his earlier forecasts underestimated progress, including advances on mathematical benchmarks. That record makes him cautious, but he still expects constraints to matter if AI cannot automate enough research before the current pace of input growth becomes harder to sustain.

Kokotajlo agrees that compute growth can slow; the disagreement is how much AI-assisted research might push in the other direction within the next few years. Greenblatt adds that chip improvements, labor and training efficiencies could continue to contribute even if one source of growth weakens.

Minsky asks about 'research taste' in the AI Futures model. Kokotajlo defines it as how much a researcher learns per experiment: choosing a decisive test, drawing the right lesson and selecting the next experiment instead of pursuing many dead ends.

Fast coding alone does not necessarily speed a large experiment when computing capacity is the bottleneck, Kokotajlo says. Better research judgment might reduce the number of expensive experiments needed. His model represents that improvement with assumptions about how AI research ability changes over time.

Greenblatt says important uncertainties remain: how much can small experiments reveal about frontier-scale training, and how much can coding labor accelerate large experiments through better implementation? The answer changes the relative importance of more work, faster work and better research judgment.

Sevilla finds detailed models informative but difficult to evaluate. If research taste is already improving, he expects some of its effect to appear in observed progress. He wants to watch both capability measures and company revenues for signs of a persistent change in the growth rate.

Greenblatt asks for earlier indicators than revenue, which can be influenced by adoption, government policy and data-handling rules. A direct measure of AI research progress could reveal acceleration sooner than waiting for it to appear in a released model or in company revenue.

Minsky asks how forecasters prevent complex models from simply encoding their prior beliefs. Kokotajlo acknowledges that additional parameters create more room for bias. He says his detailed model is primarily a way to organize competing arguments and test their interaction, not a reason to trust a forecast merely because it is elaborate.

The model can represent both a slowdown in compute scaling and a speedup from automated research, with explicit assumptions about their strength. Erdil adds that a model can expose inconsistent intuitions: once assumptions are written down, a preferred timeline may require parameter values its author finds implausible.

Greenblatt says even a moderate acceleration could produce very capable systems quickly, while oversight may not keep pace. A rapid transition could also reorganize society whether the systems are aligned or not, making preparation valuable across several possible outcomes.

Kokotajlo lists loss of control, concentrated power, geopolitical conflict, job disruption and misuse of biological capabilities among the risks he wants society to address. His policy preference is slower, more cautious development, especially avoiding an unconstrained recursive self-improvement loop.

He argues that removing people from the development loop can make both speed and control harder to manage. If a misaligned system designs its successor, he worries that the successor could inherit objectives chosen by that system rather than by people.

Asked for a longer-timeline perspective, Sevilla emphasizes improving public understanding rather than prescribing a particular policy. Epoch's role, as he describes it, is to provide measurements and analysis so others can reason about the technology and choose what to do.

He wants labs to share more information about trends that outsiders cannot currently observe. Greenblatt similarly calls for transparency that can be checked by independent groups, while saying transparency alone may not resolve the risks he sees.

Erdil says he works on making models and agent swarms behave as users intend. To him, unwanted actions reduce commercial usefulness as well as raising safety concerns. He expects that pressure to make behavioral reliability more important to companies.

He also worries that rapidly grown AI organizations lack mature processes for investigating serious incidents. He compares the need for causal investigation and prevention with established safety practices in aviation, and doubts such institutions can be built instantly.

Greenblatt distinguishes an incentive to make products commercially usable from an incentive to reach a socially acceptable level of risk. A short-term patch could allow deployment while leaving a deeper problem unsolved; he also raises concerns about security and tampering.

Erdil agrees organizational change may take time. He wants better measurements of whether research progress is accelerating sharply: the actions appropriate to a two-year transition could differ substantially from those appropriate to a slower one.

Minsky asks the panel to consider a more skeptical view. Historically, people have often overestimated how quickly a new technique would cover every domain; human and machine abilities can be uneven, and earlier AI forecasts were followed by long winters of disappointment.

The panel debates which historical comparisons are appropriate. Chess and other narrow successes did not immediately automate all cognition; several speakers nevertheless think recent broad progress across tasks is different. Minsky cautions that expert and public expectations have varied across eras.

One panelist recalls expecting the gap between a fluent conversational model and broad automation to be short. The intervening years make him give weight to the possibility that a persuasive story about automated AI research could likewise omit hard steps.

Another response is that the relevant evidence is now broader than performance at a single game. The speaker sees meaningful progress across many kinds of computer-based work, even where agents still make awkward mistakes or require workarounds.

Minsky asks why impressive capabilities are not more obvious in productivity statistics. A panelist argues that the scale of AI revenue and the time required for adoption matter: a limited aggregate effect today does not necessarily contradict rapid growth in a still-small sector.

The discussion turns to programming, where speakers see more immediate task-level benefits, and to capital expenditure as a possible leading indicator. They acknowledge that measuring effects across the wider economy is harder than examining a particular workflow.

Asked what could prove his own view wrong, Sevilla says he might underestimate how much progress can occur inside a single data center without broad economic deployment. Erdil says the next few years should provide substantial evidence about whether a software-only takeoff is possible.

Greenblatt describes two possibilities: AI research might accelerate with systems less generally capable than expected, or a long tail of hard-to-train abilities might delay full automation. Either could make a forecast based on another sector's progress misleading.

Kokotajlo raises a different uncertainty: talking publicly about an AI race and recursive improvement may help people make better decisions, but it might also encourage powerful actors to move faster. He says his work rests partly on the hope that discussion improves judgment.

Audience questions begin with which capabilities may arrive next. Sevilla points to more capable computer use, longer-running agents and robotics, while noting that robotics benchmarks are less mature and progress is difficult to measure from demonstrations alone.

Greenblatt expects tasks with short, cheap feedback loops to improve faster, because they are easier to train and verify. He also distinguishes iterative improvement on a measurable objective from generating a good new idea; the gap could make progress uneven across domains.

Asked what would make them stop their current safety or capability work, Kokotajlo says he would change focus if alignment and control problems were solved or a still more urgent problem arose. Erdil says reliability problems already draw some of his work toward alignment, and he would reconsider more sharply if measures of research acceleration changed dramatically.

An audience member asks whether open models could help defend against dangerous AI. Greenblatt sees value in openness for independent scientific study, but weighs that against possible cyber and biological risks from more widely available capabilities. He is skeptical that giving everyone another misaligned system is a sufficient defense.

Another question asks when AI systems will deceive people successfully and regularly. Panelists say the answer depends on definitions and note that models can already produce misleading behavior. They distinguish that from the more consequential scenario of an autonomous group controlling substantial computing resources without oversight.

They disagree about how soon such a self-sustaining group might be possible. The exchange is explicitly speculative: speakers consider whether systems would need to earn money, obtain access to computing through other means or overcome security and legal barriers.

The final audience question concerns public resistance, regulation and protest. Sevilla expects a substantial reaction against AI, including fear about jobs; Erdil expects social and political resistance to slow development but is uncertain by how much.

Kokotajlo welcomes additional time and argues that restrictions should address internal deployment and recursive improvement, rather than only limiting AI use in professions. Another panelist notes that some builders themselves publicly express concern, which could affect the political response. Minsky thanks the panel and audience.