Astra’s benchmark numbers changed after the blog went live

OpenAI changed the benchmark tables in its GPT-6 Astra launch post several times after it first went up on September 3, and some figures were still moving the next day, Fortune reported after comparing Internet Archive snapshots of the page. Most of the edits made Astra look better against its predecessor or against Anthropic’s models. Several were later reversed.

Astra’s own math score did not change at any point. What changed were the numbers it was being compared with, along with a handful of Astra figures.

What moved, and in which direction

The first archived snapshot is from 2:23 p.m. ET. Five snapshots up to 3:11 p.m. show the same figures. A sixth, taken at 5:20 p.m., shows five metrics changed. By the time Fortune published, some had changed back:

• Astra hallucination rate (OpenAI’s internal benchmark): 4.2% → 2% → 4.2%
• GPT-5.6 Sol hallucination rate: 12.2% → 9.4% → 12.2%
• Claude Fable 5.1 on FrontierMath Tier 4 (v2): 87.8% → 78% (snapshot at 5:17 p.m.) → 83%
• GPT-5.6 Sol on FrontierMath Tier 4 (v2): 83% → 80.5% → 83%
• GPT-5.6 Sol on OpenAI’s internal version of ExploitBench: 5.5% → 11.5%
• Astra on coding: 57.7% → 57.9%
• Claude Fable 5.1 on HealthBench Professional: 56.6% → 58.1%
• Claude Opus 5 on HealthBench Professional: 54.5% → 56.4%

Astra stayed at 97.6% on FrontierMath Tier 4 (v2) the whole time, and OpenAI puts math in the opening paragraph of the announcement. In the first snapshot Astra led Fable 5.1 by 9.8 points. After the edit the gap was 19.6 points, and now it is 14.6.

Not every edit went Astra’s way. The two HealthBench Professional changes raised Anthropic’s scores. Fortune notes that rival models’ scores usually come from published leaderboards, so OpenAI does not run those evaluations itself.

The changes also started before the post went live. In the embargoed draft OpenAI sent to Fortune and other outlets, Astra’s ARC-AGI-3 score was 98.6%. The live post shows 99.99%. “We always verify evals before publication so adjustments between draft and final version are normal,” a spokesperson said at the time.

A launch post that went up, came down, and went up again

OpenAI planned to publish the post at 2 p.m. ET. It did publish shortly after 2 p.m., then pulled it. When OpenAI’s X account shared the link at 3:32 p.m., the page returned an error. At 3:50 p.m. Sam Altman posted the link himself: “We hit a little snag getting the blog post deployed, but it is really great.” Commenters, and Fortune, still got the same error. About an hour later the page loaded normally.

OpenAI gave Fortune three explanations for the gap in turn. First it blamed a bug in its content management system, then an internet outage. Finally it said the post had been retracted for reasons it could not disclose, and that those reasons had nothing to do with the benchmark figures. The republished version carried the different numbers.

OpenAI’s account of the edits

“We care deeply about getting evaluations right,” an OpenAI spokesperson said. “Most evaluations have noise within a few percentage points based on the exact checkpoint, scaffold, and evaluation run used in reporting. For our launch blog, we made fixes to ensure the numbers represent our best estimate of available model performance, so that users can make meaningful comparisons.”

Each metric belongs to a different research team inside OpenAI. The teams calculate their numbers and pass them to a central team that publishes them. The post’s disclaimer reads “Evaluation scores are the maximum at any effort,” and footnotes add caveats to each metric. OpenAI says the published figures come from the best possible conditions and may differ from what ChatGPT users get in production.

That gap between test conditions and the shipped product is behind at least one planned reversal. OpenAI said it is looking into reverting Sol’s ExploitBench score to 5.5%, because the 11.5% result reflects a reasoning level that is not commercially available for Sol.

The harness, meaning the tooling a model gets to work through a task, matters even more. The Arc Prize Foundation, which created ARC-AGI-3, measured Astra at 99.9% when it ran with a particularly powerful harness. On the benchmark’s standard harness, the foundation’s result was 62.7%. That is still well ahead of any other publicly released model. “Things like harness, reasoning level and other factors inform evals,” OpenAI said.

“Benchmaxxing”

Anka Reuel and Mike Hardy, researchers at the Stanford Intelligent Systems Laboratory and the Stanford Trustworthy AI Lab, raised a different explanation: re-running evaluations under different conditions until the scores come out higher. The practice is common across the industry, not only at OpenAI. “This can be done in a very tight timeframe, and it’s better for their marketing,” they said. Fortune suggests re-runs could explain the 0.2-point bump in Astra’s coding score.

The two researchers also said the GPT-6 Astra system card, the technical document meant to explain how the evaluations were run, doesn’t always do that. On the internal hallucination benchmark, the one whose figures moved most, they said it gives “barely any details about the evaluation” and “doesn’t even include the number of test items.”

Disputes like this have happened before. In 2025, Meta denied reports that it had inflated Llama 4’s scores by publishing results from an internal version of the model rather than the public one. Yann LeCun, Meta’s former chief AI scientist, later admitted the company had “fudged” the results.

Vincent Sunn Chen, who leads benchmark and evaluation research at Snorkel AI, said late changes are not unusual. “It’s usually a function of final launch logistics,” he said in an email. “A benchmark score reflects a specific measurement setup: the model checkpoint, configuration (including how much time and compute the model is allowed), harness, eval/grading configuration (e.g., non-determinism in the judge). All of those are typically still shifting in the final days before a launch, so I’m not surprised that there were some updates.”

Chen also wants an industry norm: when a company revises published benchmark numbers, it should say what changed in the assessment, so researchers can tell what the new figure actually measures.