Claude Code matches Codex on SWE-Bench Pro at twice the cost

Three coding agent harnesses resolved between 44% and 52% of the same 64-task set curated from SWE-Bench Pro, while the GPU cost of one sweep on GLM-5.3-Flash ranged from $22.8 with Codex to $45 with Claude Code or Pi. The figures come from the aistack team at imec, which swapped harnesses while holding models, hardware and inference settings fixed.

At a glance

• The harness is the layer that turns a model into an agent: it manages the system prompt and context, orchestrates tools, parses output, recovers from errors and decides when a run stops.
• On GLM-5.3-Flash, Claude Code pushed 480.2M input tokens through the model against 192.1M for Codex, and on Qwen3.8-27B 445.8M against 330.5M, a gap the resolve rates do not follow.
• Codex finished the GLM-5.3-Flash sweep in 1.26 hours against 2.48 for the other two, which at $18.16 per hour on Modal turned into $22.8 instead of $45.

Harness choice is still largely made on preference and GitHub visibility, and these numbers suggest that preference stays cheap only until it meets a rented GPU. Accuracy appears to be roughly harness-independent at this scale, which moves the decision onto wall time, context volume and stability, the metrics that decide how many tasks a fixed cluster clears per hour. For teams automating pipelines rather than developer desktops, that reorders the usual priorities.

Six harness-model runs landed within three resolved tasks of each other

The benchmark reused 64 coding tasks curated from SWE-Bench Pro. On Qwen3.8-27B in FP8 on a single H200, Claude Code resolved 33 of 64, Codex 31 and Pi 28. On GLM-5.3-Flash in FP8 across four H200s, Codex resolved 33, Claude Code 32 and Pi 32.

Each developer in the evaluation framework ran as a separate harness worker with its own environment, resources and web access. Every harness ran at default settings with no customisation, and the inference engine configuration was identical across all three. GLM-5.3-Flash was previously known as "Ox alpha", while Qwen3.8-27B scores 52 on the Artificial Analysis Intelligence Index and 51 on its Agentic Index.

Codex on GLM-5.3-Flash generated 2.1M output tokens against 3.97M for Claude Code

Most harness-model combinations produced 3.33M to 3.97M output tokens per 64-task sweep, the exception being Codex on GLM-5.3-Flash at about 2.1M. That run cleared 50.5 tasks per hour against 25.8 for Claude Code and 25.9 for Pi on the same model.

On Qwen3.8-27B at $4.54 per hour, the full sweep cost about $9 through Codex, $10.16 through Pi and $10.26 through Claude Code. On the four-GPU GLM-5.3-Flash setup the same workload cost $22.8, $45 and $45. The team's earlier estimate for that workload at Anthropic API token prices was around $98.

Claude Code v2.1.233 dropped the prefix cache hit rate to 29.84%

A Claude Code version released during testing degraded time to first token, generation throughput and the prefix cache hit rate, which fell to a mean of 29.84% on Qwen3.8-27B against 97.36% on the patched release. The team logged it as issue 87227 in the anthropics/claude-code repository.

Pi produced a separate anomaly: one or two worker agents repeatedly entered a doom loop, collapsing into repetitive output until the orchestrator killed them at the four-hour limit, on both Qwen3.8-27B at xhigh and GLM-5.3-Flash at max. Pi's figures in the report are mean estimates drawn from the unaffected runs.

KV cache pressure tracked the same order: on Qwen3.8-27B at concurrency 8, peak utilisation reached 59.8% of the pool for Codex, 72.1% for Pi and 78.8% for Claude Code, with means of 22.2%, 24.6% and 32.4%. No run spent any share of its time at or above 90% of the pool.

The reruns and the next test

The team says a closer look at what the agents were actually doing forced it to rerun the experiment more than once, with that account due in a following post early next week. A comparison of inference engines is planned for a week or two after that. The report frames its numbers as a snapshot and says they will be updated as models and harnesses change.