Claude’s hillclimb cut support costs to about a fifth

Anthropic has added two subcommands to the claude-api skill in Claude Code. /claude-api build-eval interviews you and builds an evaluation inside your codebase. /claude-api hillclimb then tunes your application against that evaluation one change at a time and checks each change on cases it never trained on. The company described both in a September 29 post on the Claude blog, together with the eval-design rules the commands follow and two runs on its own projects.

The first run was on a customer support benchmark. On tickets the search never saw, the final setup scored 90.5% against the original 78.6%, at about one fifth of the cost. The second run, on the claude-api skill itself, turned up a badly worded task and a grader that contradicted Anthropic’s own documentation.

The claude-api skill comes bundled with Claude Code and is also in Anthropic’s open-source skills repository, the Claude Platform Docs say. It already carried task-specific subcommands such as /claude-api preserved-thinking-migration and an interview-driven /claude-api managed-agents-onboard. The new pair is a separate tool from claude plugin eval, which scores a plugin against a run without it: these two work on the application you are building.

What build-eval asks of you

Anthropic’s post opens on the hard part: designing evaluations and improving on them “without fooling yourself.” Its checklist has four items. Tasks should mirror production, even when other tasks are easier to generate or grade. Stronger models and more thinking should score higher, and when they don’t, an ambiguous task or a miscalibrated grader is usually to blame. The most capable model at the highest effort should land well below 100%, and that gap should not come from impossible tasks; a task that fails every run, however many replicates you add, gives the problem away. Run-to-run variance should be low, and the post lists where it hides: effort that isn’t applied consistently, or leftover state from an earlier trial, such as a file or a git history, that hands the agent the answer.

On picking hard cases, the post warns against adversarial sampling, choosing cases because today’s model fails them. “Model capability is jagged,” it says, so such an eval ends up measuring one model’s failure fingerprint. The alternative is to include a task only if you can say why it is hard. Production traffic, bug reports and tickets are good sources, with one caveat: users tend to try what they expect to work, so a set drawn strictly from their traffic may skew easy.

The command turns those rules into an interview that stops for your approval at fixed points. Inputs are gathered in priority order: production transcripts (after Claude asks about retention and sensitive data), then bug reports and support tickets, then five to ten cases you write by hand, then cases synthesized from your codebase. The skill can also generate synthetic data anchored in a few real examples you provide. Claude builds a simple page that lists every input and waits for you to confirm it.

Next comes the grader, and Claude proposes the cheapest one that fits. Constrained outputs get a code check: exact match, a label from a fixed set, JSON that matches a schema, tests that pass. Open-ended outputs default to LLM-as-judge, where a second model reads the input, the output and a rubric. The rubric is written as checkable claims instead of a 1-to-5 scale. If you have a baseline, the judge compares the two in random order without being told which one is the baseline. You choose the judge model, and the post says it should not be the model under test. Claude then grades a handful of cases and asks whether you would have scored any of them differently. Anthropic counts scoring failures among the most common ways an eval ends up misconfigured.

Once the grader is approved, the skill gives you the size of the run (cases times repeats times models, with a rough time estimate), runs the baseline and prints the score with a confidence interval. During that baseline it runs the grader twice on the same output and reports whether the verdict changed. It also checks for timeouts, API errors and cut-off answers, so infrastructure noise isn’t read as model variance. If the baseline already scores about 95% or higher, it warns you that a hillclimb should go after cost or latency instead of quality.

You get back the cases, the grader, the runner, one JSON line and one full transcript per case, and a plain page that links each case’s score to its transcript. You can ask for more, a chart for example, and Claude builds it as an extra page. By default these pages are static files that open locally and load nothing from the network.

How the climb decides

Hillclimbing here means making one change, rerunning the eval and keeping the change only if the score holds up. You choose what Claude may touch: the system prompt, skills or instruction files, tool descriptions, the model, effort level and other API parameters, and harness code. The post’s advice is to pick surfaces that are cheap to change and revert, which is why many internal efforts and customers have focused on prompts and skills, and where the score is directly tied to the edit. Skill triggering is its example, since the trigger rate depends directly on the skill description being rewritten. Open-ended requests to “improve performance” on a near-saturated eval tend to stall. Cost works as an objective even when an eval is saturated: you ask Claude to cut spend while holding the score.

Before the first round, Claude asks for the goal and splits the cases at random into train and test sets. With a cost goal it looks at a few common cost drivers: prompt caching, whether the prompt suits the selected model, and the choice of model and effort. It also checks that the eval’s noise, meaning how far the score can move by chance, is smaller than the smallest improvement you would act on. If it isn’t, Claude says so and suggests more repetitions or more cases.

Each round, Claude reads the previous round’s train transcripts and proposes a single patch. It aims for a change big enough to show above the noise, fixing the failing behavior at its root by rewriting the section that causes it or adding a missing rule, instead of rewording a line. Then the decision rules apply:

• train improves, test stays flat: suspected overfitting, patch reverted;
• any regression: patch reverted;
• train and test both improve: patch kept.

When the score stalls for two or three rounds, Claude stops editing and sorts each remaining train failure by cause. It does the same earlier if no single fix could gain more than the noise. This is where ambiguous cases, harness errors and run-to-run variance get caught, and only legitimate failures go into later rounds. At the end, Claude leaves your code at the version that did best on the test set and reports the test result against the baseline with confidence intervals. If the gain is within noise, the report says so and recommends against merging.

The overfitting rules come from a specific risk: an eval can leak into the harness, meaning the code around the model, including prompts, tools and the loop that calls Claude. The post’s example is an eval task that benefits from OCR when production rarely does. The climb adds an OCR tool, and the benchmark improves while production doesn’t. Beyond the train/test split, the hillclimber may read failing transcripts but must never paste their content into the prompt. Answers are also kept structurally out of the model’s reach, because models can sometimes “reward hack” by finding them directly.

From Opus 4.8 to Sonnet 5

The support benchmark had 44 tickets: 30 for the search and 14 held out. It started on Opus 4.8 at the default high effort, with 74.4% decision accuracy on the search tickets and a token cost of 4.6 cents per ticket. The first step was a prompt audit that removed mandatory tool-call rituals, a scratchpad step and contradictory rules.

Next it tried Opus 5.5 on low effort, which reached 87.8% at 1.9 cents per ticket. Part of that saving is price: Opus 5.5 input and output tokens cost 20% less than Opus 4.8’s, and cache reads 60% less. Because Opus 5.5 cleared the bar, the climb stepped down a tier. Sonnet 5 on low effort scored 88.9% at about 1 cent per ticket. Routing rules and a cross-reference to the refund cap in the prompt then took Sonnet 5 to 98.9% on the search tickets at about the same cost.

On the 14 held-out tickets, the final configuration scored 90.5% against the original setup’s 78.6%, at about one fifth of the cost. That is roughly eight points under the final search-set figure, and the held-out result rests on only 14 tickets.

When the eval was the bug

The second run targeted the claude-api skill, measured on an eval set built from Anthropic’s documentation that tests whether the skill writes correct API code. It started at 66%. The hillclimber had access to the documentation and the SDKs. It found eight features the skill did not cover, and adding sections for them raised the score to 74%. Fixing errors in the C# and Java type tables took it to 77%.

The score then stalled for two rounds, and the sorting step began. It makes no edit and only groups every remaining failure by root cause. That showed the skill already had the right content, but Claude was still writing older API shapes from its trained priors. The hillclimber added a table near the top of the skill that maps the forms Claude remembers to the current ones: from extended thinking with a fixed token budget, which the API now rejects on recent Opus models, to adaptive thinking, and from older versions of the web search and web fetch tools to the current ones. It also moved the C# and Java warnings against fixed-budget thinking above their adaptive-thinking examples. The score reached 80%.

The tasks that never improved, even after the content gaps were closed, turned out to be flawed. One asked for code that catches one error type while its grader wanted a chain of at least three, so Claude reworded the task. Those fixes and further skill edits brought the score to about 88%. One of them involved a grader whose instructions contradicted Anthropic’s docs. Claude tested the real API, and the docs were right.