Context compression cut 39% of tokens and saved almost nothing

One developer ran headroom against SR SWE bench in Harbor: 25 tasks, three branches, plain Claude versus two compression modes, token mode and cache mode. The tasks were picked to favor the tool and everything was billed at Anthropic's API rates. Input tokens dropped from 215M to 132M, a 39% cut. The money came out to $0.37 per task with a negative median, and 13 of the 25 tasks got more expensive.

The cache explains it. On the baseline branch roughly 98% of input tokens were served from cache at $0.30 per million, against $3.00 for fresh ones. Because compression rewrites the history, token mode invalidated the cache 123 times versus 14 for cache mode, pushing 6,062,098 tokens to full price. The lost discount: $16.37, against $15.41 saved.

headroom's own counter overstated the savings by about 1.9x. Solve rates barely moved: 5 of 25 for token mode, 4 each for the baseline and cache mode.