Claude Code matches Codex on SWE-Bench Pro at twice the cost

On GLM-5.3-Flash across four H200s, a 64-task coding sweep cost $22.8 with Codex as the harness and $45 with Claude Code or Pi, though all three resolved 32 or 33 of the tasks. imec’s aistack team published these results in a September 10 post on Imec-int.

Accuracy barely changed. Across both models tested, the three harnesses resolved between 28 and 33 of the 64 tasks, which the authors call a variation of two to three tasks. The cost gap comes from how many tokens each harness pushes through the model, and so from how long each session keeps a GPU busy.

The setup

The harness is the layer you actually install and run. It decides which tools the model can call (terminal, file editor, test runner, web search), manages the context, formats prompts, parses output, recovers from errors and decides when to stop. The aistack team’s earlier self-hosting study kept Claude Code fixed as the harness and changed the hardware. This time the hardware stayed the same and only the harness changed. All three harnesses ran on their default settings with no customisation, and the inference engine configuration was identical for all of them.

The tasks are the same 64 the team curated for that first study. They come from SWE-Bench Pro, which Scale describes as 1,865 problems from 41 actively maintained repositories. Each simulated developer is a separate harness worker with its own environment, resources and web access. The two models were Qwen3.8-27B in FP8 on a single H200 ($4.54/hr on Modal), which scores 52 on the Artificial Analysis Intelligence Index, and GLM-5.3-Flash in FP8 on 4×H200 ($18.16/hr). GLM-5.3-Flash was previously known as “Ox alpha”.

From output tokens to the invoice

The chain runs through output tokens. Generating them takes much longer than processing input, so output volume accounts for a large part of wall time. A session that generates more holds its GPU slot longer, and the same hardware resolves fewer tasks per hour. Five of the six harness and model pairs generated between 3.33M and 3.97M output tokens per sweep. Codex on GLM-5.3-Flash generated 2.1M.

Wall time followed. On GLM-5.3-Flash, Codex finished the sweep in 1.26 hours, while Claude Code and Pi each took 2.48. That works out to 50.5 tasks per hour for Codex against 25.8 for Claude Code and 25.9 for Pi. Counting only resolved tasks, Codex managed 26 per hour and the other two 12.9 each. Rented or owned, the GPUs run the whole time, whether the model is reasoning or a tool call is executing, so the cost of a sweep is its wall time multiplied by the hourly rate.

On Qwen3.8-27B with one H200 the spread is narrow: about $9 for Codex, $10.26 for Claude Code and $10.16 for Pi. The authors add that all of these still cost considerably less than paying API prices. In their earlier post, the same workload came to around $98 at Anthropic’s API token prices.

Where the extra context goes

Input tokens tell the same story. On Qwen3.8-27B, Claude Code pushed 445.8M input tokens through the model against 330.5M for Codex, and resolved 33 tasks against 31. On GLM-5.3-Flash the gap widens to 480.2M against 192.1M, and there Claude Code resolved one task fewer than Codex. Pi landed at 348.5M and 366.4M. Claude Code’s context volume has come up before: in July, a separate test found that Claude Code sends about 33k tokens before it reads the prompt, against roughly 7k for OpenCode.

More context also means a bigger KV cache, the GPU memory where the inference engine keeps each live session’s attention state. The team measured it on Qwen3.8-27B with eight concurrent workers. Mean utilisation was 22.2% of the pool for Codex, 24.6% for Pi and 32.4% for Claude Code. Peaks were 59.8%, 72.1% and 78.8%. None of the three spent any part of the run at 90% or above, so the pressure matters once you push concurrency or throughput higher.

One update, a third of the cache

Prefix caching lets the engine reuse the KV cache it already computed for the unchanging start of a conversation, so it does not recompute it on every turn of a long agent trajectory. It only works while that start stays unchanged. Put something dynamic near the top, usually in the system prompt (a date, a UUID that keeps changing), and nothing after that point can be served from cache.

The team says this is exactly what happened with Claude Code v2.1.233, a release that came out during testing. After the update, time to first token, generation throughput and prefix cache hit rate all got worse. On Qwen3.8-27B the mean hit rate fell to 29.84%, with p10 to p90 at 23.4% to 37.8%. On the patched release it was 97.36%, with p10 to p90 at 95.9% to 98.6%. The problem is tracked in an issue on the claude-code repository. Apart from that version, the harnesses barely differed: mean hit rates for every harness and model pair fell between 94.17% and 97.36%. All the Claude Code numbers in the comparison come from the patched release.

Pi’s doom loops

Pi had a different problem: one or two workers kept getting stuck in what the authors call a doom loop. It happened on both Qwen3.8-27B (at xhigh reasoning) and GLM-5.3-Flash (at max), on different tasks each time. The models eventually collapsed into repetitive gibberish, and the orchestrator killed the workers when they hit the four-hour limit. In the log the team published, one worker repeats git’s housekeeping notice over and over: “Auto packing the repository in background for optimum performance. See “git help gc” for manual housekeeping.”

The team assumes a fix is on its way. The Pi figures in the comparison are based on mean estimates and on the runs where the loop did not appear. The authors add that the affected runs would push total wall time up significantly.

What should self-hosters do?

On accuracy alone, the authors keep their advice short: “If you are picking a stack based on resolve rate alone, save yourself the benchmarking and analysis paralysis – at least for now.” They expect most teams won’t want to change the harness their developers use every week, because the harness affects developer experience. The harness behind automated workflows is easier to change. They also call the Codex and GLM-5.3-Flash result a snapshot that next month’s models and harnesses could change. And when they looked at what the agents were actually doing, they found behaviour that made them rerun the whole experiment more than once. That write-up is due in a follow-up post.

The version anomalies are what they tell self-hosters to monitor: “blindly updating to newer versions can considerably impact your invoice at the end of the month, and not just by a few %.”