A harness swap took Astra from 62.7% to 99.9%

OpenAI’s GPT-6 Astra now has the top ARC-AGI-3 score under both of ARC Prize’s test harnesses, and which harness you look at changes how the news reads. In a September 3 post on the Arcprize blog, the benchmark’s organizers report 62.7% on the Semi-Private set under their Standard harness. Under a Provider Adapter harness, which keeps OpenAI’s own context handling switched on, the score is 99.9%.

ARC-AGI-3 is the series’ first interactive benchmark. It drops an agent into abstract turn-based games with no instructions. The agent has to explore, work out what counts as winning, build a model of how the environment behaves and plan its moves. The environments rely only on core knowledge priors, the basic notions of objects, space and counting that people bring to any puzzle. Their difficulty was calibrated on human test-takers, who solve all of them. In March, the ARC-AGI-3 paper put frontier AI systems below 1%.

What the harness changes

A harness is the code that sits between the model and the game. It decides what the model sees on each turn and what carries over to the next one. ARC Prize’s Standard harness is deliberately minimal and provider-neutral, and every model runs through it the same way. It supplies everything needed to solve each game, but the model has to decide for itself which notes to keep in view as it plays. Under it, Astra at max reasoning effort scored 62.7% for $26,098.

The Provider Adapter harness adds two things OpenAI built for Astra. It preserves the model’s opaque reasoning state between requests, which ARC Prize says it cannot see. It also uses compaction, condensing older turns so a long session still fits in context. With that setup Astra at high effort reached 99.9% for $18,817. Across the Public and Semi-Private sets and all reasoning levels, Provider Adapter runs were about 3.66x faster by total recorded elapsed time, and they used 49% fewer tokens across the 167 game-and-reasoning-level pairs that both harnesses solved.

ARC Prize frames the two setups as answers to different questions. The Standard harness asks how models compare under one shared interface, and the organizers set the bar there: “We believe a future AGI should be able to solve ARC-AGI-3 under these conditions.” The Provider Adapter asks how far a model gets with the context tools its vendor designed for it. From now on both results go on the ARC-AGI leaderboard, each clearly labeled, and the open-source testing repository and testing policy cover both approaches. That leaves 62.7% as the only Astra figure measured under the interface every other model runs through. Near-perfect scores with heavier scaffolding had also shown up before: in August, NVIDIA reported its AVO coding agent clearing all 183 levels.

Cheaper at max effort

Astra’s cost curve runs against the usual pattern. ARC Prize says higher reasoning levels generally cost less because Astra solves games in fewer actions, which means fewer model calls and fewer tokens overall. Here is the full grid:

• max: 62.7% for $26,098 (Standard), 98.6% for $17,332 (Provider Adapter)
• xhigh: 59.3% for $37,317 (Standard), 98.4% for $18,147 (Provider Adapter)
• high: 54.8% for $40,705 (Standard), 99.9% for $18,817 (Provider Adapter)
• medium: 38.6% for $48,090 (Standard), 98.4% for $19,285 (Provider Adapter)
• low: 17.5% for $38,166 (Standard), 98.0% for $21,298 (Provider Adapter)
• none: 35.2% for $49,791 (Standard), 96.7% for $23,457 (Provider Adapter)

Under the Standard harness, max effort cost less than low effort and scored far higher. The curve isn’t smooth, though. With reasoning switched off, Astra scored 35.2%, above its low-effort result, and that was also the most expensive Standard run. Under the Provider Adapter every level scored 96.7% or better, the best score came at high effort rather than max, and the most expensive Adapter run still cost less than the cheapest Standard one.

ARC Prize also gives a human price for comparison. Participants in its controlled testing were paid $115 per 90-minute session plus $5 per completed game. They attempted about nine games per session, roughly $12.78 per attempted game before bonuses. The organizers point out that most of that fee pays for people’s time and willingness to sit the test. If you price only the brain’s energy as electricity (20 W at $0.20 per kWh), it drops to about 0.6 cents per session, or 0.067 cents per attempted game.

Fewer moves than people

Before launch, ARC Prize tested about 500 members of the public, who were not selected for puzzle-solving experience or ability. It set each level’s human baseline at the median action count among the players who finished that level. The ARC-AGI-3 paper describes the benchmark’s scoring as efficiency-based and grounded in those human action baselines.

In the Provider Adapter harness, Astra at max effort used fewer actions than the human baseline on 96.0% of levels, and 51.7% fewer actions per level on average. “This is a material milestone,” the organizers write. By the benchmark’s own measure, Astra matched human action efficiency and then went past it.

The organizers had expected the opposite. Before launch they assumed action efficiency would keep separating humans from AI: even when a model solved an environment, it would need far more exploratory moves than a person. They say that still holds for brute-force approaches, but frontier models behave in a more all-or-nothing way. “Once frontier AI ‘understands’ the mechanics, it generally executes within the range of human efficiency.” Most benchmarks track only cost efficiency, meaning the compute spent. Action efficiency measures how much experience with an environment a system needed before it could act well.

Notes in its own shorthand

The replays show how Astra got there. Under the Standard harness the model chooses which strategy notes to carry forward. Astra tracked objects, coordinates, rules and unfinished plans in a compact notation it made up for each environment. ARC Prize says other models have done something similar, but Astra’s notes stood out for their precision and information density. The organizers call it an on-the-fly algebraic shorthand rather than a full programming language. Some lines from the game s5i5:

• L8: hub q2 (8↓). Lengths: 14=1… records the level, a local rotation index and the lengths of the game’s mechanisms
• extend8 to3; retract10 to2; shorten8 to1 is an ordered plan of changes to the color-8 and color-10 mechanisms
• 9−=(39,4), rotate=(49,18), 14+=(59,11) maps each operation to the coordinates of the control that performs it

In another game, wa30, a single line, Turn 5: P=(24,20), empty, facing west, packs together a turn counter, the player’s position, what it is carrying and which way it faces.

ARC Prize also ran Astra in PRO-LONG, a harness from one of its early ARC-AGI-3 red-teaming partners that gives the model a sandbox for running its own code. There Astra built a set of tools for each game: board parsers, game-state models, search algorithms, planners and persistent notes. On longer runs it wrote small game-specific libraries. In tu93, a maze game with guards and moving patrols, it began with navigation in maze_solver.py, added combat rules in combat_solver.py, modeled the patrols in patrol_solver.py and used sync_state.py to check its predictions against what the game actually showed. ARC Prize notes that its human testers had no code interpreter or scratch pad, so PRO-LONG results should be read as the model and its tools combined. A footnote adds: “No evidence of trying to break out of the sandbox was observed.”

Not AGI, by ARC Prize’s own definition

The organizers call Astra “a noticeable step-function change in frontier model capabilities” and say it clears the bar ARC-AGI-3 set: building causal world models efficiently and reaching goals without instructions. They also repeat what they said at launch, that saturating the benchmark would not prove AGI. They define AGI as a system that can learn any skill a human can, as efficiently as a human. “We are not claiming that it is AGI,” the post says.

Part of the reason is scope. ARC-AGI-3 environments have deterministic, closed-ended mechanics and goals, and the organizers say they do not represent the complexity and open-endedness of the real world. For the next generation of benchmarks, ARC Prize says it is looking at how to evaluate recursive self-improvement and open-ended innovation.