ponytail’s token savings: 10.3% off the bill, not the promised 20%

JetBrains ran 80 paired tasks against the ponytail skill for Claude Code, the third token-saving add-on in its testing series. The pitch: 54% less code, 22% fewer tokens, 20% lower cost, 27% less time. The median measured result: 15% less code, 10.3% lower cost (p=0.004), and 11% less time.

Setup was Claude Code 2.1.201 with claude-sonnet-5 at medium reasoning effort, using the SkillsBench benchmark. The savings show up where the agent overcomplicates things: code dropped 31% on large builds, while the median didn’t budge on tasks that were already short.

One big caveat on installation. Dropping SKILL.md into the skills folder wasn’t enough: the model never invoked it once across 10 sessions. Every number here came from a branch where the ruleset is force-injected by a SessionStart hook.

On quality, 65 tasks scored the same, 9 got worse, and 6 got better. The full evaluation run cost $246.09.