rtk promised 60-90% fewer tokens. JetBrains measured 7.6% more spend

rtk (“Rust Token Killer”) is a Claude Code add-on that uses a PreToolUse hook to intercept shell commands and hand the model a compressed version of the output. The pitch: 60-90% token savings. JetBrains ran a paired A/B test to check it. Setup: Claude Code 2.1.201, claude-sonnet-5, the SkillsBench benchmark, 86 tasks.

At low reasoning effort, rtk came out 7.6% more expensive per task at the median (p=0.004); at high effort the difference was roughly zero. Output quality matched across both branches. Why: the hook sees only about a third of Bash calls and under 20% of output characters, while most of the cost comes from cached re-reads that are billed at one tenth the price.

Meanwhile rtk’s own analytics claimed 96.2 million tokens saved, measured against the full raw output as its baseline. An earlier entry in the series tested the caveman skill: promised -65%, measured -8.5%.