METR clocks Anthropic benchmark gains at 1.9x old pace

METR spent August 12 to October 29 inside Anthropic with access to internal training and benchmark data. On November 5 METR reported that since October 2025, scores on Anthropic’s internal capability index have been growing at 1.9x their earlier rate (95% confidence interval 1.5-2.8x). Only the 1.9x was measured. The broader claim that Anthropic now improves its models on well-specified tasks at about twice the old pace, and the guess that roughly 80% of today’s speedup comes from Anthropic’s own use of AI, are METR’s subjective judgement, and much of the evidence behind them was redacted by Anthropic.

What 1.9x measures

The index is AECI, Anthropic’s version of the Epoch Capabilities Index, which combines many benchmark scores into one number. METR defines acceleration as a ratio of growth rates. At 2x, one year brings two years of progress at the old pace: an index that used to gain 15 points a year would gain 30. By that measure, Anthropic made as much benchmark progress in the past year as between October 2023 and October 2025, roughly the span from Claude 2.1 to Sonnet 4.5. That is the number behind the “nearly twice as fast” in Reuters’ headline.

The Haiku 5.5 system card reported 1.7x. METR’s figure is higher mainly because it included data on newer models that have not been released. When METR changed which benchmarks and models went in, how models were dated and how saturated benchmarks were handled, the rate moved between 1.1x and 2.9x. METR says it would be surprised if an improved version of AECI showed no acceleration, or acceleration above 3x.

The headline conclusion leans more on judgement than the index does. Most METR staff on the project believe Anthropic is improving its models on well-specified tasks at around twice the rate of March 2024 to October 2025. They would not be surprised by 1.3x or 3x, they are less sure how this carries over to broader, less well-specified work, and the team had internal disagreements. METR also says the jump in AECI could come from overfitting to the benchmark distribution, or could be an artifact of how the index is built. In the other direction, AECI data points lag the research and are fairly sparse. Redacted evidence made METR take the roughly 2x reading more seriously. It also gave some indication that the rate could be above 2x and rising, though METR would be surprised if it were significantly higher.

How much of it is AI?

METR ruled out one cause with confidence: training compute did not start growing faster. On the other candidates it examined (hiring, better data, algorithmic discoveries, other bottlenecks being cleared), it found no decisive evidence, and Anthropic’s own staff did not agree. METR thinks the acceleration in late 2025 was likely influenced by non-AI factors, but most details of those factors were redacted. For October 2026, it guesses that a majority, perhaps around 80%, of the acceleration is due to AI, and that AI’s share has grown over time. Staff described dramatic uplift from AI in interviews and surveys, which METR calls challenging to interpret.

METR was told the gains come mainly from AI assisting researchers rather than from fully autonomous contributions, though there are some signs autonomous work will play a larger role soon. Anthropic has already published autonomous work of this kind: in September, Sonnet 5 ran the post-training of an early Opus 4.8 checkpoint and reached safety scores approaching those of production Opus 4.8.

What Anthropic redacted

Anthropic gave METR what METR calls unprecedented access. It included pretraining and RL metrics, training compute for each domain, benchmark scores for internal models, AI speedup analyses, forecasts for a future model, and interviews with staff. Anthropic also had the right to redact any non-public information, and it removed a majority of the evidence METR found most informative. Anthropic said the redactions were “aimed at removing confidential information that could accelerate other AI developers.” One of the redacted items is METR’s detailed estimate of how much AI has sped up AECI progress and each stage of the training pipeline, both cumulatively and at the current moment. As a result, METR says, the public has to rely on its judgment “much more than would be ideal.”

METR names how it could be wrong in each direction. It could be overestimating the acceleration if it gave too little weight to non-AI factors, to AECI’s focus on tasks with verifiable answers, or to signs that Anthropic’s growth in 2023-2025 was unusually slow, which would make the comparison baseline low. It could be underestimating it if some growth rates have risen further recently, if more AI R&D tasks get automated, or if AECI has understated real-world impact.

Room to compound

The worry METR set out to test is compounding: AI speeds up R&D, which produces more capable models, which speed up R&D further. METR found no evidence of significant bottlenecks over the next 12 months from limits on algorithmic progress, training compute, experiment compute or data. But its view of those bottlenecks changed several times as more detailed information came in, and it says it is not confident in it. Its conclusion is that “there is a significant risk that further capabilities progress could lead to even faster acceleration over the coming year.”