Harness swap doubled GPU cost at the same resolve rate

Swapping the agent harness barely changed how many coding tasks got solved, yet it doubled the GPU bill. imec’s aistack team ran Claude Code, Codex and Pi over 64 tasks curated from SWE-Bench Pro, each harness on default settings with an identical inference engine config.

All three landed in a 44–53% resolve band. On GLM-5.3-Flash across 4 H200s, the Codex sweep cost $22.8 in rented GPU time, while Claude Code and Pi ran $45 each. Both pushed more context through the model and held the GPUs longer.

Two anomalies hit the runs. Claude Code v2.1.233 broke prefix caching and dropped the mean hit rate to 29.84% from 97.36% on the patched release, and Pi workers repeatedly fell into doom loops that the orchestrator had to kill.

The caching regression is filed as issue #87227 in Anthropic’s claude-code repository.