Skip to content

OpenAI

Same model, better scaffolding: 13.3% to 38.3% on ARC-AGI-3

4 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: openai.com

OpenAI’s developer guide to GPT-5.6, published August 13, has GPT-5.6 Sol climbing from 13.3% to 38.3% on ARC-AGI-3, on roughly 6x fewer output tokens, once the model’s reasoning was kept between turns and long conversations were compacted. Both are controls in the Responses API, so the jump came from configuration alone on the same model, in a run OpenAI did itself.

Retaining reasoning on GPT-5.6 comes down to passing the previous response ID in the Responses API, which then carries the reasoning across tool calls and turns, OpenAI explains in its write-up of the ARC-AGI-3 harness investigation. Compaction is a separate setting that summarizes older context rather than dropping it outright.

What the official harness threw away

The official ARC-AGI-3 harness discarded all of the model’s private reasoning after each game action, leaving Sol a record of past moves and brief notes but not the plans or insights behind them, so it had to work the game out anew with every move. Context limits were handled by rolling truncation: past 175,000 characters, the oldest messages were discarded, so the model lost earlier observations and actions and spent much of the task with a fuller context window, which OpenAI says can slightly impair performance. With reasoning retained, Sol spent less time thinking before each action, since it no longer had to interpret the game from scratch every turn, and got much better at learning over time and holding a coherent strategy.

The ARC Prize Foundation answered OpenAI’s post by defending its verified scores, which deliberately use a “no harness” setup: every model gets the same observations, system prompt and action limits, and conversation state is kept on the client side, OfficeChai reported. ARC said it is working with OpenAI and other labs on how server-side state management might be folded into verified testing without giving any one provider an edge. The 38.3% rests on reasoning the Responses API keeps between calls, the server-side kind of state the verified runs leave out.

Luna and Terra for the volume work

On Agents’ Last Exam, GPT-5.6 Sol at “low” reasoning outperformed GPT-5.5 at “high” with the harness held constant, and startups in OpenAI’s production testing report significant cost improvements across a range of workflows after lowering reasoning effort from the prior defaults. On BrowseComp, a search benchmark built around obscure facts, GPT-5.5 (Extra High) scored 84.36% for a total of $33.27 three months before the guide, while GPT-5.6 Luna (Extra High) scored 84.04% at launch for $1.33: about a third of a point lower at roughly a 25th of the cost. OpenAI has cut prices again since.

Until the 5.6 family, a flagship model at the highest available reasoning was the best option for long-horizon work, largely because flagships were significantly more capable than cost-optimized models at long contexts and tool calling. With more test-time compute, Luna and Terra can now often perform similarly to GPT-5.4 and GPT-5.5 while costing significantly less. The guide fits them to high-volume workloads, latency-sensitive interactions and repeated steps inside agent workflows. Its example is a legal-tech startup that parses handwritten memos before an agent analyzes them: Terra or Luna takes the extraction step, and the guide counts significant savings from no longer running a frontier model through the whole use case.

Orchestration and the prompt cache

Multi-agent orchestration is native in the Responses API once enabled: a primary agent delegates tasks to subagents, they pursue them in parallel, and their output returns to the primary agent for the final synthesis. ChatGPT’s ultra setting works the same way. GPT-5.6 has a strong sense of how many subagents to spawn and when, in OpenAI’s account, but the behavior is very steerable: instructing the model on when to invoke subagents makes it likelier to spawn them only where the extra tokens buy better performance.

Programmatic tool calling moves deterministic work into code. GPT-5.6 writes JavaScript that orchestrates tools, runs independent calls in parallel and processes their output outside the context window, which reserves model tokens for judgment and cuts cost, latency and context rot. In the guide’s case, an agent retrieves 100 filings, filters them by date and identifies the relevant transactions without the model reasoning over every intermediate result in its context.

Across the whole model family, the prompt cache’s time to live is now at least 30 minutes, and cache breakpoints can be set deterministically within the context window, which the guide credits with significantly better cache hit rates at startups. Alongside the breakpoints, an appropriate prompt_cache_key makes it more likely that a request lands on the inference engine that already served the same prefix, which cuts latency.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.