Harness swap doubled GPU cost at the same resolve rate

Three coding agent harnesses resolved almost the same number of tasks on the same benchmark while costing twice as much in GPU time: on GLM-5.3-Flash across four H200s, a Codex sweep cost $22.8 against $45 each for Claude Code and Pi. The figures come from a harness comparison published by imec's aistack team, which ran 64 tasks curated from SWE-Bench Pro through each harness on default settings.

At a glance

• The harness is the layer around the model that manages system prompts, context, tool calls and error recovery; all three ran on default settings with an identical inference engine configuration.
• On GLM-5.3-Flash, Codex pushed 192.1M input tokens and generated 2.1M output tokens over the sweep, while Claude Code pushed 480.2M input and generated 3.97M output for one fewer task resolved.
• Two version-level anomalies distorted runs: Claude Code v2.1.233 dropped the mean prefix cache hit rate to 29.84% from 97.36% on the patched release, and Pi workers hit doom loops the orchestrator killed at four hours.

For teams running their own GPUs, the finding shifts the optimisation target. Resolve rate is what benchmark tables advertise, but the harness appears to determine how long a task occupies a GPU slot, and that is what shows up on the invoice. The doubling on GLM-5.3-Flash also reads as a warning about version upgrades: a caching regression inside a harness release can move monthly infrastructure spend more than a model swap would.

Resolve rates across the six combinations spanned 28 to 33 tasks out of 64

On Qwen3.8-27B in FP8 on a single H200, Claude Code resolved 33 of 64 tasks, Codex 31 and Pi 28. On GLM-5.3-Flash in FP8 across four H200s, Codex resolved 33, while Claude Code and Pi each resolved 32. The spread across all six combinations is two to three tasks.

Qwen3.8-27B scores 52 on the Artificial Analysis Intelligence Index and 51 on the same firm's Agentic Index. The benchmark set was curated from SWE-Bench Pro, and each developer in the evaluation framework ran as a separate harness worker with its own environment and web access.

On Qwen3.8-27B, Claude Code pushed 445.8M input tokens against 330.5M for Codex, roughly 1.35 times the context for one more task resolved. Output token counts over the same sweep were closer: 3.72M for Claude Code and Pi, and 3.33M for Codex.

Codex on GLM-5.3-Flash finished the sweep in 1.26 hours against 2.48 for Claude Code

Codex on GLM-5.3-Flash completed the 64-task sweep in 1.26 hours of wall time, against 2.48 hours for both Claude Code and Pi on the same model. That translated into 50.5 tasks per hour and 26.0 resolved tasks per hour for Codex, against 25.8 and 12.9 for Claude Code.

At the $18.16 per hour Modal rate for four H200s, the Codex sweep cost $22.8 while Claude Code and Pi each cost $45. On the single H200 at $4.54 per hour, the gap narrowed: $9 for Codex, $10.16 for Pi and $10.26 for Claude Code.

The team previously measured the same workload at around $98 using Anthropic API token pricing. GPU cost here is wall time multiplied by the hourly rate, since the hardware stays rented whether the model is generating tokens or a tool call is running.

Claude Code held 32.4% of the KV cache pool on average against 22.2% for Codex

With the same inference engine configuration on Qwen3.8-27B at concurrency 8, mean KV cache utilisation was 22.2% for Codex, 24.6% for Pi and 32.4% for Claude Code. Peaks reached 59.8%, 72.1% and 78.8% respectively; none of the runs spent any time at or above 90%.

Prefix cache hit rates were close across harnesses on the patched builds: 96.58% for Codex, 95.50% for Pi and 97.36% for Claude Code on Qwen3.8-27B. Version 2.1.233 of Claude Code broke caching, and the mean hit rate fell to 29.84%, with time to first token and generation throughput degrading alongside it. The regression is filed as issue #87227 in Anthropic's claude-code repository.

One or two Pi workers repeatedly got stuck in doom loops, on different tasks each time and with both models. The output collapsed into repetition until the orchestrator killed the run at its four-hour limit; Pi's figures are mean estimates from the clean runs.

Why the experiment was rerun

The aistack team says a closer look at what the agents were actually doing turned up behaviour that forced a full rerun of the experiment, and that the findings will be published in the next post. A comparison of inference engines is slated after that. No exact publication dates are given, and the team notes both harnesses and models move fast enough that the current cost leader may not hold.