anthropic
Prompt audit in Claude Code cuts cost 14.6%
Promtime
anthropicAnthropic reports that running its new /claude-api prompt-audit command on six legacy prompts cut costs by 14.6% and raised accuracy by 5.3% on average when a customer support benchmark was migrated from Claude Opus 4.8 to Opus 5. The guidance is published on the Claude blog and shipped inside the claude-api skill for Claude Code.
At a glance
- Prompt caching stores a request's prefill state so repeated prefixes are read back cheaply; a miss rewrites the prefix at 1.25 times the normal input price, or twice that for a one-hour cache.
- On FrontierCode Diamond's hardest 50 tasks, Claude Fable 5 scores 11.5% at low effort for $5.35 per task and 30.9% at max effort for $19.00, roughly 2.7 times the score for 3.5 times the cost.
- A second command, cost-optimize, profiles token spend and then applies caching, batching, output bounds and effort changes; across four public benchmarks starting from Sonnet 5 it lowered cost by between roughly 52% and 73%.
Cost and performance are usually traded against each other, and the framing here is different: the waste sits in prompt scaffolding written for older models, not in the model choice. That reads as an argument for treating prompts as versioned artifacts that get re-audited on every migration, since instructions added to patch a weaker model's failures are the ones a frontier model now follows literally and bills for.
Removing planted anti-patterns cut spend 14.6% and lifted accuracy 5.3%
Anthropic started from a clean prompt and planted one anti-pattern at a time, giving six legacy prompts: a retired thinking setting, contradictory refund rules, a manual scratchpad, "verify twice", "be maximally thorough", and a mandatory six-step procedure. Each ran on Opus 4.8, on Opus 5 with only the model ID changed, and on Opus 5 after a single audit pass.
On Opus 5 the verification ritual duplicated an order lookup on every refund, and the thoroughness booster turned into dozens of unneeded knowledge-base searches. Removing them eliminated the extra tool calls and duplicated reasoning that drove the spend, according to Anthropic.
Accuracy rose for three reasons: the retired thinking setting made the API reject every routing request outright; the contradictory refund rules led Opus 5 to withhold four refunds it owed; and the manual scratchpad collided with built-in thinking, so on three tickets the model wrote a tool call inside its reasoning and never executed it.
Fable 5.1 at low effort matches Fable 5 at high effort on CursorBench 3.2
Effort tells Claude how hard to work: at low effort it reaches conclusions faster, at high effort it deliberates, verifies and explores alternatives. On Fable 5.1, Humanity's Last Exam without tools scores about 53% at low effort for about $0.30 per question and about 61% at max effort for about $2.23, where the last step adds roughly half a point for 46% more cost.
On CursorBench 3.2, Fable 5.1 at low effort matches Fable 5 at high effort for a third of the cost, helped by prompt-cache reads priced at $0.25 per million tokens against $1.00 on Fable 5; at Fable 5's prices the low-effort configuration would still cost about 40% less. A flat cost-performance curve across effort levels indicates a task that is not bound by thinking compute.
On SWE-bench Verified, cost-optimize cut cost 55% and median steps from 29 to 17
Across four public benchmarks starting from Sonnet 5, cost-optimize cut LegalBench cost by about 58%, with thinking tokens falling from 102,779 to 8,284; tau2-bench retail by about 73% with pass rate flat via explicit cache breakpoints; and OfficeQA Pro by about 52%, from $136.20 to $64.87, using batch processing and document caching.
On SWE-bench Verified the default configuration already cached correctly, so savings of about 55% came from medium effort and output constrained to a few concise sentences, with median steps per task falling from 29 to 17 and prompt tokens from 75.2M to 33.7M.
A third command, /claude-api hillclimb, searches over model, effort and prompt. Starting from Opus 4.8 at high effort on a customer support benchmark, it landed on Sonnet 5 at low effort with added routing rules and a refund-cap cross-reference, reaching 98.9% train accuracy at 1 cent per ticket and 90.5% on 14 held-out tickets against 78.6% for the original setup, at about one fifth the cost.
Which command to run first
Anthropic recommends prompt-audit immediately after migrating to a frontier model; it scans prompts, skills and tool descriptions in the working directory for anti-patterns, covering both application code that calls the Claude API and Claude Code's own configuration files such as CLAUDE.md.
cost-optimize measures effort and model trade-offs only when an evaluation is supplied, and hillclimb requires one as well, scoring its final configuration on held-out data. Anthropic points to its own documentation and a cookbook for the remaining levers, which include batching unattended work.
Comments
No comments yet. Be the first.
Join the conversation
Sign in with Google to leave a comment. Your name and avatar come from your Google profile, and the comment appears after moderation.
We only use your name and avatar from Google. We never store your email address.
