Skip to content

Anthropic

METR clocks Anthropic benchmark gains at 1.9x old pace

5 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: zglg.work

Since October 2025, Anthropic’s models have been gaining on its internal benchmark index 1.9 times faster than before, METR found in an assessment published November 5 after 11 weeks inside the company. The 1.9x is a measured trend on AECI, Anthropic’s version of the Epoch Capabilities Index, and it includes models the public has not seen. The claim that AI drives most of it is METR’s own guess, resting largely on evidence Anthropic redacted, and when we reran the same method on public scores for Anthropic’s released models, we got 1.44x.

From August 12 to October 29, in an engagement that grew from four weeks to 11, METR staff looked at pretraining and RL metrics, training compute by domain, benchmark scores for internal models, internal speedup analyses, forecasts for a future model and interviews with staff, METR writes in its report. Anthropic could redact any non-public information and did so for a majority of the evidence METR found most informative, saying the redactions were “aimed at removing confidential information that could accelerate other AI developers.” METR says the public therefore has to rely on its judgment “much more than would be ideal.” That judgment: most of the team believes Anthropic is likely improving its models on well-specified tasks at around twice its March 2024 to October 2025 rate, and would not be surprised by 1.3x or 3x. The Reuters headline, “nearly twice as fast with help from AI”, puts the measured trend and the attribution in one phrase.

What the 1.9x measures

The figure (95% confidence interval 1.5–2.8x) is the ratio of the AECI slope after an estimated breakpoint on October 7, 2025 to the slope before it. Models are dated by when most Anthropic employees could use them. METR says that choice generally shows slightly more acceleration, because Anthropic now holds recent models longer before release. The interval covers uncertainty in each model’s scores only, not which models went into the fit.

Varying the snapshots, the benchmarks or the regulariser keeps the result between 1.9x and 2.2x. The wide 1.1x–2.9x spread METR reports comes from one choice, the link function (the curve that turns benchmark scores into an underlying ability level), and the alternatives fit almost equally well, with out-of-sample R² between 86.1% and 87.2%.

Adding other labs’ frontier models to the fit cuts the acceleration to 1.4x. METR gives two readings. If Anthropic’s models were subfrontier in 2025, its earlier progress was understated, and “1.4x is more accurate”; if they are less benchmark-optimized than rivals, the 1.9x is closer to right.

Anthropic’s own estimates have moved over six weeks. The Opus 5.5 system card on September 22 fitted a 1.53x slope change (95% range 1.20 to 1.82), found a one-time jump the better fit in 99 of 100 resampled fits, and concluded that “under either reading the slope has not doubled.” The 1.70x METR cites from the Haiku 5.5 card of October 7 sits in a chart legend, beside a fit with a one-time jump at Claude Mythos Preview. METR now says newer internal models make a permanent slope change look more plausible than that jump. Each card refits the AECI scale, so the three numbers are not on one scale.

How much of it is AI?

METR is confident the acceleration did not come from faster growth in training compute. Beyond that, it says Anthropic’s own staff did not agree and it found no decisive evidence. Its guess is that a majority, “perhaps ≈80%”, of the October 2026 acceleration is due to AI, while non-AI factors likely played a large role in late 2025; its detailed estimates of AI’s effect were redacted.

Staff self-reports in October 2026 Slack polls reached a 12x speedup, which METR calls “only very loose evidence” given the low-effort answers. It was told R&D is sped up mainly through AI assistance rather than fully autonomous work, with some signs that autonomous contributions will play a larger role soon.

For the next 12 months, METR saw no significant bottleneck in algorithmic progress, training compute, experiment compute or data, and so sees a significant risk of even faster acceleration over the coming year. Its picture of the bottlenecks changed several times as more detailed data arrived, and it says it is not confident in it.

Our analysis

What we checked

On November 6 we downloaded Epoch AI’s public ECI scores and applied METR’s method, the ratio of least-squares slopes after and before October 7, 2025, to Anthropic’s released models with their public release dates. Each run below is one specification; we did not search for the one that maximises or minimises the ratio.

How we check stories

Anthropic’s released frontier went from 13.8 to 19.8 ECI points a year, a 1.44x acceleration, while the frontier across all labs held its pace at 0.99x. Our inference is that this looks like Anthropic catching up to a frontier whose slope stayed flat, the first of METR’s two readings. Our 1.44x lands next to METR’s 1.4x, but it comes from a different index and model set.

SpecificationAcceleration95% interval
Anthropic frontier, released models1.44x1.13–1.85
Same, baseline from January 20241.28x1.06–1.54
All 22 released Anthropic models, no frontier filter1.10x0.95–1.29
Frontier across all labs0.99x0.82–1.18
METR, internal AECI1.9x1.5–2.8

Our runs use public release dates, which METR says tend to show less acceleration than its internal dating, so some part of the gap between 1.44x and 1.9x, which we did not quantify, comes from dating.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.