Reasoning off, Astra still hits 96.7% on ARC-AGI-3

GPT-6 Astra scored 99.9% on ARC-AGI-3, the result OpenAI used to declare the AGI era. That run used OpenAI’s own Provider Adapter. ARC Prize also ran the same model through its standard harness at maximum reasoning, and there it scored 62.7%. Both figures appear in the table ARC Prize published on September 3, the day Astra launched, and The Next Web set them side by side in a September 9 analysis.

The weights were identical in both runs. Only the harness changed: the software around the model that decides which tools it can call, what it keeps from one request to the next and how its context is trimmed.

Two ways to run the same model

ARC-AGI-3 is a set of turn-based games. The agent sees a screen, takes an action, sees the result and has to work out the goal without any instructions. Scoring is based on action efficiency against human players, so wandering, backtracking and brute force all cost points. ARC Prize says a 100% score means an agent “can beat every game as efficiently as humans.”

ARC Prize’s standard harness gives every model the same minimal interface. Between turns the model carries forward only the notes it chooses to write. OpenAI’s Provider Adapter keeps the model’s opaque reasoning state from one request to the next, so the hidden reasoning from earlier turns is still available on the next move, and it compacts conversations once they grow long.

Under the standard harness at maximum reasoning, Astra scored 62.7% and the run cost $26,098. Under the adapter at high reasoning it scored 99.9% for $18,817. The higher score was also about $7,300 cheaper. On the 167 game-reasoning pairs that both setups solved, ARC Prize measured the adapter runs at 49% fewer tokens and roughly 3.66 times faster.

One row stands out. With reasoning effort set to none inside the adapter, Astra still scored 96.7%, which is 34 points above the same model at maximum reasoning in the standard harness. On this benchmark, changing the wrapper moved the score more than turning reasoning all the way up.

The comparison that travelled

The comparison that spread after launch put Astra’s 99.9% against GPT-5.6 Sol’s 7.8%. Those two numbers come from different harnesses: Sol’s was measured in the standard harness. The Next Web says its own launch coverage ran the pair without that caveat, and that a follow-up piece on the AGI claim described human parity on the same basis. Measured the same way, Sol at 7.8% against Astra at 62.7% is a jump of about 55 points. That is enormous, but well short of the 92 points the headline pairing implied. The benchmark’s own arXiv paper, submitted in March, reported frontier systems scoring “below 1%.”

ARC Prize did not draw OpenAI’s conclusion. The foundation said it is “not claiming that it is AGI”, and co-founder Mike Knoop wrote that “we lack evidence to call this AGI yet”. ARC Prize’s figures are not fully consistent either: François Chollet gave the standard-harness result as 66% in a post, while the published table says 62.7%. The table on the blog is the primary record.

Numbers that kept moving

Emily Forlini of Fortune compared archived snapshots of OpenAI’s launch post and found five metrics altered after publication. Astra’s hallucination rate was 4.2% in the first snapshot, 2% by 5.20pm, and is back at 4.2% now. Anthropic’s Fable 5.1 fell on FrontierMath from 87.8% to 78% and then settled at 83%. Sol’s ExploitBench score doubled from 5.5% to 11.5%, a result that reflects a reasoning level Sol does not offer commercially. OpenAI told Fortune it is investigating whether to revert that figure.

The pre-publication draft sent to media under embargo put Astra’s ARC-AGI-3 score at 98.6%. The live post says 99.99%. The edits did not all favour Astra, since two Anthropic scores on HealthBench Professional went up, but most of them did. OpenAI also took the post down after publishing it and then restored it, and said it could not disclose why. The company’s explanation to Fortune was that most evaluations carry a few percentage points of noise, depending on checkpoint, scaffold and evaluation run.

Anka Reuel and Mike Hardy, Stanford researchers at the Intelligent Systems Laboratory and the Trustworthy AI Lab, call the practice benchmaxxing: re-running evaluations under different conditions until the number improves. In Astra’s system card they found “barely any details about the evaluation” for the internal hallucination benchmark, adding that it “doesn’t even include the number of test items”. Vincent Sunn Chen of Snorkel AI reads the edits less harshly. He told Fortune that scores routinely shift in the last hours before a launch, while checkpoint, configuration, harness and grading are all still changing. His proposal is a norm that companies say what changed whenever they revise a published figure. There is precedent for the harsher reading. In 2025, critics accused Meta of publishing Llama 4 results from an internal build rather than the public one, and Yann LeCun later said the company had fudged them.

What’s actually on sale

Artificial Analysis ran its own tests, and the results are less dramatic. On its Coding Agent Index, Astra scores 67 in Codex, level with Claude Opus 5 and Fable 5, while Fable 5.1 leads at 70. On its Intelligence Index, Astra scores 61. That equals the model it replaces, puts it five points behind Fable 5.1 and also behind Meta’s Muse Spark 1.3. Astra costs $10 per million input tokens and $50 per million output tokens, two and a half times Sol’s price, and Artificial Analysis calculates it at 75% more per task at maximum effort.

The firm did find real gains in some areas. Hallucination on its knowledge benchmark fell from 92% to 51%, and long-horizon knowledge work rose by around 80 Elo. But an economically weighted task benchmark fell by roughly the same amount, and Astra regressed on banking support, scientific Python and long-context reasoning. A day later Artificial Analysis released version 4.2 of its index, with harder tasks and more private test sets that it says exist to prevent gaming.

The adapter is not secret tooling. It relies on documented API features that any developer can call, and harnesses are now products in their own right: Anthropic, Google and Microsoft each sell one with its own pricing. Nvidia built a harness that took Claude Opus 5 from 30.2% on ARC-AGI-3 to clearing every level. Still, the system that scored 99.9% was model plus adapter, and what OpenAI sells is the model. ARC Prize says that from now on it will publish results from both harnesses side by side.