Claude Code writes the eval, grader included

Anthropic has given the claude-api skill in Claude Code two commands for eval work. /claude-api build-eval writes a runnable eval for your application: a set of inputs, a way to run the app on each one, and a grader that scores the results. /claude-api hillclimb then improves the app against that eval, one change per round. The commands were announced on September 28 by Anthropic’s ClaudeDevs account on X, alongside a claude.dev post on how to design evals and how to improve scores on them without fooling yourself.

build-eval collects inputs in a fixed order of preference. It starts with production transcripts, after asking about retention and sensitive data, then moves to bug reports and support tickets, then 5–10 hand-written cases. Synthetic cases come last, and only when anchored in a few real examples. It produces a review page for the inputs and waits for you to confirm. For grading, it uses a code check where the output is constrained, and a second model scoring each answer against checkable claims where many answers are valid. Claude grades a handful of cases and asks whether you would have scored any of them differently. It also runs the grader twice on the same output to see whether the verdict changes. Nothing paid runs until you have signed off on the inputs, the grading method and the cost.

The guidance behind this lists what a good eval needs: tasks that mirror production, scores that rise with stronger models and more thinking, low run-to-run variance, and headroom, meaning the most capable model at the highest effort should score well below 100%. Anthropic warns against keeping only the cases today’s model fails. An eval built that way can end up measuring that one model’s failure fingerprint. Raw user traffic has its own problem: users try what they expect to work, so it can skew easy.

In hillclimb, you decide what Claude may change (the system prompt, tool descriptions, the model, the effort level, harness code) and whether the goal is better results or lower cost at the same quality. Cases are split at random into a training set and a held-out test set. A change is reverted if it causes a regression, or if the training score rises while the test score stays flat.

Anthropic’s customer-support example used 44 tickets, 14 of them held out. It started on Opus 4.8 at high effort, at 4.6 cents a ticket. Claude removed contradictory rules and mandatory tool calls from the prompt, moved to Opus 5.5, and ended on Sonnet 5, both at low effort. The 98.9% that headlines the post was measured on the 30 tickets the search could see. On the 14 held-out tickets the score was 90.5%, up from 78.6% for the original setup. That held-out result is the one to quote, and it is still a gain of about 12 points, at about one fifth of the cost, or under a cent a ticket.

The last step is a check on the result itself. If the final gain is within noise, hillclimb recommends against merging it.