METR Finds Faster AI Research at Anthropic, With Major Uncertainty

Matthew Berman says AI-assisted development may be producing a self-reinforcing increase in the pace of AI research. He bases the discussion on an independent assessment by METR, an AI evaluation organization, of research and development at Anthropic.

METR worked with Anthropic and reviewed internal material, researcher interviews and metrics. Berman initially describes the access as close to unrestricted, then distinguishes it from a separate program offering employee-like access: this assessment was subject to Anthropic’s approval of information requests.

The report asks whether model capability progress has accelerated, how much of that acceleration is attributable to AI, and whether bottlenecks could slow further progress. Berman says the answers make him concerned about the pace of future development.

Anthropic could approve or deny requests, and it had rights to redact non-public information before publication. Berman emphasizes that limitation before discussing the findings. METR says much of the evidence it found most informative was redacted.

Most participating METR staff estimate that, as of October 2026, Anthropic’s models were improving on well-specified tasks at around twice the rate of the March 2024–October 2025 period. They attach substantial uncertainty: a rate closer to 1.3 or 3 times the earlier pace would not surprise them.

Berman links the measure to research work such as writing code and building the systems around model development. METR’s estimate concerns the rate of capability progress on well-specified tasks, rather than a claim that every capability suddenly doubled.

The assessment says other factors probably played a large role in the acceleration of late 2025. For the current acceleration, METR guesses that a majority—perhaps about 80%—is due to AI, while explicitly describing that judgment as substantially uncertain.

Berman shows METR’s illustrative chart comparing an earlier trend with a hypothetical path without AI and the observed path. The chart is labeled as a sketch using fake data; its purpose is to explain the attribution, not to display measured values.

He connects the timing to the improvement in AI coding tools around late 2025, saying developers experienced a change in what models could do. In his account, assistance with large codebases and features can feed back into the lab’s own development process.

Berman’s concern is compounding: a better model may help researchers make the next model more capable, improve the use of computing resources and amplify human work. He connects that possibility to arguments about an intelligence explosion.

He discusses reported gains in compute efficiency and cites Google’s AlphaEvolve work as an example of AI contributing to engineering improvements. He argues that even a small efficiency gain can matter at the scale of a major AI company.

METR says it did not identify clear bottlenecks or steep diminishing returns that would prevent further compounding acceleration in the near term. Berman interprets this as a warning about rapid progress. METR also says its understanding changed during the assessment and it may be missing important bottlenecks.

The report says Anthropic appears to have made roughly as much benchmark progress in the preceding year as in the two years from October 2023 to October 2025. Berman relates that finding to the faster cadence of model releases he has observed.

The chart he displays uses AECI, Anthropic’s version of the Epoch Capabilities Index. Berman points to a steeper trend after late 2025 and the contribution of newer internal models. METR estimates a 1.9-fold change in the benchmark trend, with uncertainty about how it translates to broader tasks.

The assessors say they received convincing evidence that the acceleration was not caused by an acceleration in the scale of training compute. Berman notes that Anthropic redacted further details, limiting what the public can examine directly.

METR also considered interviews, researchers’ estimates of their own productivity, headcount, code and experiment usage. Berman acknowledges that casual self-reports are not a controlled scientific measurement, but sees them as one signal alongside the other evidence.

Among the anonymous comments he reads, researchers describe faster work and say coding has become less of a constraint. One describes ideas and compute as remaining bottlenecks. Berman highlights the potential importance of finding promising research directions, even when implementation becomes faster.

The report says AI is not yet making highly significant, fully autonomous contributions to the most important parts of the model-training process. Researchers nevertheless described early signs that more autonomous contributions could become possible. Berman connects that observation to recent AI work on mathematical proofs.

He speculates that with sufficient compute, models might test many possible directions even without human-like intuition, although intuition could reduce the cost of searching. METR’s own assessment says its conclusions about future bottlenecks remain uncertain.

Berman returns to the project’s setup. METR used about 2.5 full-time-equivalent staff and began the engagement in mid-August. The group requested information it considered relevant, while Anthropic could accept or reject requests and the parties negotiated over sensitive data.

The project was not the employee-like access arrangement described in Anthropic’s October 15 update. Berman stresses the distinction because the earlier arrangement gave Anthropic control over both access to material and redaction of non-public details.

The original scope included checking how Anthropic allocated computing resources, including between capabilities and safety work. Anthropic changed the scope midway, removing that verification before METR reached substantive conclusions. Anthropic said it wanted to focus on risk assessment and revisit verification later.

Berman speculates about whether the company’s plans for an initial public offering influenced that decision. He presents this as his own interpretation of the changed scope, not an explanation established by the assessors.

In the appendices, Berman points to results from unreleased internal models that METR considered when assessing whether progress reflected a lasting change in trend rather than a one-time jump. He also notes that benchmarks must become harder as models approach their limits.

He closes by returning to the possibility that better models could accelerate their own development and to his concern about people losing the ability to understand the systems. He mentions research on models writing weights for other models as another example he wants viewers to consider.