anthropic

"Verify twice" wastes tokens on Opus 5, Anthropic says

Claude News

anthropic

Anthropic has packaged its cost-reduction guidance for the Claude Platform into three Claude Code commands in the claude-api skill, and the first of them, /claude-api prompt-audit, cut cost by 14.6% and raised accuracy by 5.3% on a customer support benchmark. The figures and the guidance were published on the Claude blog.

At a glance

  • The audit targets instructions written to patch older models: verification rituals, emphasis boosters, mandatory scratchpad steps, stale few-shot examples, contradictory rules and configuration such as retired manual thinking budgets.
  • A second command, /claude-api cost-optimize, profiles where token spend goes and lowered cost between 52% and 73% on four public benchmarks run from a Sonnet 5 baseline.
  • The third, /claude-api hillclimb, splits an evaluation into train and test sets; on 14 held-out support tickets its final configuration scored 90.5% against the original setup's 78.6%, at about a fifth of the cost.

Prompts tend to outlive the models they were written for, and the audit results suggest the scaffolding accumulated around older generations is now a direct cost line rather than a safety net. The three commands read as an attempt to make cost tuning a repeatable step in the upgrade path, rather than a one-off exercise for teams with an evaluation harness already in place.

Removing six planted anti-patterns cut cost 14.6% and lifted accuracy 5.3%

The test migrated a customer support workload from Opus 4.8 to Opus 5. Anthropic started from a clean prompt and planted one anti-pattern at a time, a retired thinking setting, a pair of contradictory refund rules, a manual scratchpad, "verify twice", "be maximally thorough" and a mandatory six-step procedure, producing six legacy prompts.

Each prompt was run on Opus 4.8, on Opus 5 with only the model ID changed, and on Opus 5 after a single prompt-audit pass; the 14.6% cost reduction and 5.3% accuracy gain are averages across the six. Cost fell because extra tool calls and duplicated reasoning disappeared.

Accuracy moved for three reasons: the retired thinking setting made the API reject every routing request, the contradictory refund rules led Opus 5 to withhold four refunds it owed, and the manual scratchpad collided with built-in thinking, so on three tickets Claude wrote a tool call inside its reasoning and never executed it.

A rewritten cache is billed at 1.25x the normal input price

Prompt caching stores the internal state Claude builds during prefill, and reads are billed at a fraction of full input price. The cache is pinned to one model, the prefix must be byte-exact, and the five-minute time-to-live counts from the start of the request.

When an agent blocks on a tool call that outlives the TTL, the next turn rewrites the cache at 1.25x the normal input price, or 2x on a one-hour cache. Volatile timestamps, reordered tool definitions and mid-conversation effort changes break the prefix too, although Opus 5 and Fable 5.1 accept effort updates mid-conversation without a miss.

Anthropic's fixes: mark rarely used tools defer_loading so they stay out of the cached prefix until tool search pulls them in, add system instructions as messages instead of editing the system prompt, let automatic caching move the breakpoint to the last cacheable block, and pre-warm with a max_tokens: 0 request. Claude Console diagnostics show where two requests diverged.

Fable 5 costs $5.35 per task at low effort and $19.00 at max on FrontierCode Diamond

On FrontierCode Diamond, the hardest 50 tasks, Fable 5 scores 11.5% at low effort for $5.35 per task and 30.9% at max effort for $19.00. Anthropic warns that miscalibration runs both ways: high effort can produce over-thinking, low effort stops before enough evidence is gathered.

On CursorBench 3.2, Fable 5.1 at low effort matches Fable 5 at high effort for a third of the cost. Part of that is pricing: Fable 5.1 cache reads cost $0.25 per million tokens against $1.00 for Fable 5, and at Fable 5's prices the low-effort run would still cost about 40% less.

Run from a Sonnet 5 baseline, cost-optimize cut spend by about 58% on LegalBench, where thinking tokens fell from 102,779 to 8,284, about 73% on tau2-bench retail, about 52% on OfficeQA Pro, from $136.20 to $64.87, and about 55% on SWE-bench Verified, where medium effort and shorter agent output took median steps per task from 29 to 17.

Which command fits which case

Anthropic positions /claude-api prompt-audit as the first step after a migration to a frontier model: it scans prompts, skills and tool descriptions in the working directory, including CLAUDE.md. /claude-api cost-optimize is for a full spend audit, drawing on organization usage reports through an Admin API key, the usage object on responses, or an estimate read from request-building code. /claude-api hillclimb needs an evaluation to split into train and test sets.

Comments

No comments yet. Be the first.

Join the conversation

Sign in with Google to leave a comment. Your name and avatar come from your Google profile, and the comment appears after moderation.

We only use your name and avatar from Google. We never store your email address.