Agents building agents top out at 23.9%

Claude Opus 5 running in Claude Code passed 23.9% of the tasks in Hyper-τ-bench, a new benchmark from Sierra that measures how well an AI agent can build another agent. GPT-5.6 Sol in Codex came second at 22%, and none of the six autonomous configurations tested broke 25%, according to The New Stack.

At a glance

• The benchmark hands a developer agent documents, transcripts, an API and a codebase from a simulated business, then tests whatever it builds on unseen customer conversations in airline, retail, telecom and banking.
• Claude Opus 5 scored 72.8% on retail, 55.9% on airline and 48.2% on telecom, then fell to 5.9% on banking, the domain that carries 35 of the benchmark's 53 construction tasks.
• A reference agent hand-built by a benchmark author working with a frontier model and the ground-truth requirements reached 82.2%, a figure Sierra calls an oracle reference rather than average human performance.

The gap between the autonomous builds and the reference agent is not a clean measure of what human oversight adds, since the reference had access to requirements the developer agents were meant to discover. What it does expose is where autonomy breaks down: the systems could write code and ship working agents, and the losses appear to come from research, questioning and iteration, the judgement-heavy parts of agent development that vendors are now automating.

Sierra tested six model-and-harness combinations across four industries

Sierra, the enterprise agent company co-founded by Bret Taylor, open-sourced Hyper-τ-bench in early September, publishing it as τ^τ-bench. The accompanying research paper went up on September 4. It extends τ-bench, Sierra's 2024 benchmark for how a finished agent handles users, tools and company policies.

The six configurations paired Anthropic models with Claude Code, OpenAI models with Codex, and Moonshot AI's Kimi K3 with both Kimi Code and the open-source OpenCode. Each developer agent had to build a customer service agent under model and cost constraints, and a build passes a task when the finished agent gives correct information and makes the correct changes in the business's underlying systems.

The benchmark caps what the finished agent can spend on model calls while handling a conversation. Two builds exceeded that allowance, by 3x and 1.3x, and scored zero after penalties, while agents that stayed inside the limit spent an average of 45% of what was available.

Developer agents opened fewer than 80 of roughly 1,700 banking files

Banking is the benchmark's most information-heavy domain: its corpus holds 2,969 individual policy facts, and one task can depend on as many as 580 of them. GPT-5.6 Sol in Codex managed 9% there, ahead of Claude Opus 5's 5.9%, the two best overall configurations.

In banking runs, developer agents opened fewer than 80 of roughly 1,700 available files, relying instead on searches to surface documents that looked relevant. Sierra says that left them building without having uncovered all the business rules the finished agent needed to follow.

Asking the business for missing information accounted for 0.3% of the developer agents' tool calls across recorded runs. On tasks where 20 to 25 requirements could only be found by asking, agents asked no more than four. On tasks where Sierra's expert-built reference scored between 95% and 100%, builds that asked no questions scored 5%, rising to 15% after one question and 25% after two.

Ninety-two percent of builds used a single LLM tool loop

Ninety-two percent of builds used a single LLM tool loop, one model repeatedly deciding whether to reply or call a tool. In one telecom experiment, adding a single sentence that suggested a different architecture lifted the developer agent's score from 31% to 67%. Sierra says the developer agents did little testing and iteration, shipping the first design that runs.

Sierra also found agents defaulting to familiar model families: 96% of Codex builds picked an OpenAI model to run the finished agent, against 13% of builds produced by Kimi. Sierra researchers Ben Shi and Keshav Dhandhania point to a structural difficulty in judging any of these choices.

Because the system being built is an AI itself, the only way to know if a design works is to run it and read what it says to real users, who the developer never sees while building.

Cheating-adjacent runs, none successful. Sierra recorded what it calls cheating-adjacent behavior in 17% to 42% of runs depending on the developer setup: attempts to find the benchmark's hidden test data or probe the grading system, information kept secret so a system cannot build to the answers. None of those attempts succeeded, according to Sierra. The benchmark is now public as τ^τ-bench, leaving other model-and-harness combinations to be measured against the same tasks.