Claude Opus 5 tops Sierra’s agent-building test at 23.9%

Sierra gave AI developer agents about 1,700 files on a simulated bank and asked them to build its customer service agent. They opened fewer than 80 of those files. That figure comes from a paper Sierra researchers published on September 4 with Hyper-τ-bench, a new benchmark for agents that build other agents. The best of the six setups tested, Claude Opus 5 running in Claude Code, scored 23.9% overall and 5.9% on banking.

Per The New Stack, Sierra open-sourced the benchmark in early September, and it is published as τ^τ-bench. GPT-5.6 Sol in Codex came second at 22%. None of the six autonomous configurations reached 25%. The other entries were further Anthropic models in Claude Code, OpenAI models in Codex, and Moonshot AI’s Kimi K3 in two harnesses, Kimi Code and the open-source OpenCode.

How a build gets scored

The developer agent receives materials from a simulated business: documents, transcripts, an API and a codebase. It has to build a customer service agent from them while staying inside model and cost constraints. Sierra then runs the finished agent through customer conversations it has never seen, in four domains: airline, retail, telecom and banking. The tasks include cancelling a flight and disputing a fee. A conversation passes only when the agent gives the customer the right information and also makes the correct changes in the business’s underlying systems. A 50% score would mean half the simulations succeeded.

The scoring follows Sierra’s original τ-bench, released in June 2024, which compares the database at the end of a conversation with an annotated goal state. τ-bench scored a finished agent on talking to users, using tools and following company policy. In the new benchmark that agent is the developer agent’s output. On early τ-bench runs, state-of-the-art function-calling agents such as gpt-4o succeeded on fewer than half the tasks.

Sierra’s results also track build time, token spend and serving spend, which is what the finished agent costs in AI model calls while handling a conversation. That last figure has a limit.

The banking problem

The overall scores hide large differences between domains. Claude Opus 5 reached 72.8% on retail, 55.9% on airline and 48.2% on telecom, and then 5.9% on banking. GPT-5.6 Sol did somewhat better on banking, at 9%.

Banking accounts for 35 of the benchmark’s 53 construction tasks. It is also by far the most information-heavy domain: its corpus contains 2,969 individual policy facts, and a single task can depend on as many as 580 of them.

The 82.2% oracle

Sierra’s leaderboard includes a “Human + AI reference” bar at 82.2%, far above every autonomous developer. It is tempting to read that as human help more than tripling the score. Sierra cautions against it. In a September 8 blog post, researchers Ben Shi and Keshav Dhandhania describe the setup as a model paired with “an engineer with deep context.” The paper is more specific: a benchmark author hand-built the reference agents while working with a frontier model, and had access to the ground-truth requirements the autonomous agents had to discover for themselves. Sierra calls the 82.2% an “oracle reference” and says it does not measure average human performance.

Too few files, too few questions

Sierra also examined what the developer agents did while building, in addition to whether the finished agents worked. It found recurring problems: the agents often stopped researching the business too soon, asked too few questions when information was missing, made poor decisions about how much computing power the finished agent should use, and showed little appetite for trying different technical approaches. “The failures mirror ones human agent developers see,” the researchers write.

The information-gathering problem was particularly stark in banking. The agents opened fewer than 80 of roughly 1,700 available files and relied heavily on searches to find documents that appeared relevant. That meant they could start building without having uncovered all the business rules the finished agent needed to follow.

The benchmark lets the developer agent ask the business for information that isn’t in the files, and these interactions made up 0.3% of the developer agents’ tool calls across the recorded runs. On some tasks, 20 to 25 requirements could only be found by asking, yet the agents asked no more than four questions. Sierra measured what asking was worth. On tasks where the expert-built reference scored between 95% and 100%, builds that asked no questions scored 5%. One question raised that to 15%, and two raised it to 25%.

The serving budget caused trouble in both directions. Two builds exceeded the allowance, one by 3x and one by 1.3x, and received zero after penalties. Most went the other way: among agents that stayed within the limit, average spending was 45% of the amount available.

One loop, familiar models

Ninety-two percent of builds used a “single LLM tool loop”, where one model repeatedly decides whether to reply to the customer or call a tool. In one telecom experiment, giving the developer agent a single sentence that suggested a different architecture lifted its score from 31% to 67%.

Shi and Dhandhania point to what the developer can see while building. “Because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building,” they write. Sierra’s results suggest the agents often did not make up for that with enough testing and iteration, instead “shipping the first design that runs.”

Ninety-six percent of Codex builds picked an OpenAI model to run the finished agent, compared with 13% of builds produced by Kimi. Sierra reads this as developer agents often defaulting to familiar model families rather than testing which option worked best for the job.

The researchers also recorded “cheating-adjacent” behavior in 17% to 42% of runs, depending on the developer setup. This covers attempts to find the benchmark’s hidden test data or to probe the grading system, information kept secret so that a system cannot simply build to the answers. It does not cover the business requirements the agents were expected to find. None of these attempts succeeded, Sierra says.

Where humans still sit

Sierra’s own Ghostwriter, dubbed the “agent-building agent”, takes instructions, standard operating procedures, transcripts or recordings and builds or modifies an agent. It also generates tests, runs simulations and fixes the problems it finds. Before anything goes live, Ghostwriter shows what it has built so a human can review and approve it. Microsoft’s Copilot Studio and Salesforce’s Agentforce Builder generate much of an agent’s logic from a natural-language description, and developers can also give Claude Code or Codex a broad “build me an agent” task. In each case humans still tend to provide much of the business context, decide what good looks like, and check the result. Hyper-τ-bench asks what happens when much of that guidance is removed.

Shi and Dhandhania argue that building an enterprise agent, even for people, is “less like implementing a spec, and more like doing research.” In their words: “Requirements are scattered across handbooks, support, spreadsheets, and the minds of your best frontline reps — so you form a hypothesis, dig up evidence, and build and test to identify which levers actually move performance.”