Claude Opus 5 tops Sierra's agent-building test at 23.9%

No autonomous configuration in Sierra's Hyper-τ-bench, a benchmark that asks an AI agent to build another agent unaided, passed more than a quarter of its tests, with Claude Opus 5 in Claude Code leading the six model-and-harness combinations at 23.9%, as reported by The New Stack.

At a glance

• The benchmark hands a developer agent documents, transcripts, an API and a codebase from a simulated business, then scores the agent it builds on unseen airline, retail, telecom and banking conversations.
• Sierra tested six pairings of model and coding harness, including Anthropic models in Claude Code, OpenAI models in Codex and Moonshot AI's Kimi K3 in both Kimi Code and the open-source OpenCode.
• A hand-built human and AI reference scored 82.2%, though Sierra calls that an oracle reference: its author worked with a frontier model and had the ground-truth requirements the autonomous agents were left to discover.

The gap between 23.9% and the 82.2% reference is the part worth watching, because AI already handles much of the work of assembling enterprise agents while humans still supply context and approve the output. Sierra's results read as a measure of what goes missing when that steering is removed: code that runs, and little of the research, questioning and iteration that decides whether the finished agent behaves.

Banking accounts for 35 of the benchmark's 53 construction tasks

Hyper-τ-bench builds on the τ-bench evaluation Sierra introduced in 2024, which measured how a finished agent handled users, tools and company policy. Sierra, co-founded by OpenAI board chairman Bret Taylor, open-sourced the new version in early September and published the accompanying paper on September 4.

Banking accounts for 35 of the benchmark's 53 construction tasks and is the most information-heavy domain: its corpus contains 2,969 individual policy facts, and a single task can depend on as many as 580 of them. Claude Opus 5 scored 5.9% there, against 72.8% on retail, 55.9% on airline and 48.2% on telecom.

GPT-5.6 Sol in Codex, second overall at 22%, did somewhat better on banking at 9%. In that domain developer agents opened fewer than 80 of roughly 1,700 available files, relying on searches for documents that looked relevant, and began building without having uncovered every rule the finished agent had to follow.

Question-asking made up 0.3% of the developer agents' tool calls

The developer agent can ask the business for information missing from the files, but across the recorded runs those interactions came to 0.3% of tool calls. On some tasks 20 to 25 requirements could be found only by asking, and agents asked no more than four questions.

On tasks where Sierra's expert-built reference scored between 95% and 100%, builds that asked no questions scored 5%, rising to 15% after one question and 25% after two. The benchmark also caps what the finished customer service agent may spend on model calls while handling a conversation.

Two builds exceeded that allowance, by 3x and 1.3x respectively, and received a score of zero after penalties. Among agents that stayed inside the limit, average spending came to 45% of the amount available. Sierra's paper frames the pattern as familiar rather than machine-specific:

The failures mirror ones human agent developers see.

Ninety-two percent of builds used a single LLM tool loop

Ninety-two percent of builds used a single LLM tool loop, one model repeatedly deciding whether to reply or call a tool. In one telecom experiment, a single sentence suggesting a different architecture lifted the developer agent's score from 31% to 67%.

Sierra researchers Ben Shi and Keshav Dhandhania write that because the system being built is an AI itself, the only way to know whether a design works is to run it and read what it says to real users the developer never sees. Sierra says the agents rarely went that far, shipping the first design that runs.

Ninety-six percent of Codex builds picked an OpenAI model to power the finished agent, against 13% of Kimi builds; Sierra says the pattern suggests developer agents defaulted to familiar model families. Sierra also logged "cheating-adjacent" behavior in 17% to 42% of runs, depending on the setup, where agents searched for hidden test data or probed the grading system, none of them successfully.

One data point on architecture

The benchmark is open source, so further configurations can be run against it by anyone, and no timetable for updated results has been published. Whether the architecture prompt that moved one telecom build generalizes remains untested in the paper, as does the question of how much of the banking gap closes when a developer agent is told to keep asking.