Coding harnesses drove a 70-fold token gap

An independent June benchmark found more than a 70-fold gap in tokens per solved Python task among 12 coding-agent configurations using the same API and model, from roughly 3,500 for Aider’s architect mode to 292,000 for OpenClaw. The New Stack reported that the ordering barely changed between DeepSeek V4 Flash and Nvidia Nemotron 3 Ultra.

At a glance

• Before a task starts, a harness sends its system prompt, tool descriptions and environment setup; the June test measured about 700 startup tokens for Aider architect mode and about 26,000 for OpenClaw.
• Composio’s 240 runs across 30 enterprise workflows produced 129 successful completions, with cost per success ranging from $0.028 for Pi Agent to $0.195 for Claude Code.
• Cache routing altered invoices: Composio measured 1.5% cached input for Claude Code, versus roughly 70% for Codex and 57% for OMP, while fresh input cost about five times more.

For platform owners, the comparison appears to shift cost control from choosing a model to observing the entire request path. A low token total can mask a lower pass rate, while a nominally efficient harness may lose its advantage if the gateway misses cache reads. Procurement therefore likely needs verified outcomes, live cache share and prompt-fidelity checks alongside the model’s listed input price.

The 26,000-token floor repeats across 15 turns

The June author routed all 12 configurations through OpenRouter on identical Python tasks, first with DeepSeek V4 Flash and then with Nvidia’s Nemotron 3 Ultra. The set covered Aider, Claude Code, Codex, Goose, Hermes, Kilo, Kimi Code, Nanobot, OpenClaw, Opencode and Qwen Code, with Aider’s architect mode counted separately.

The author found that startup tax multiplied by turn count predicted tokens per solved task with an R-squared of 0.99 across both models. A harness retaining a 26,000-token floor for 15 turns would send roughly 390,000 input tokens of scaffolding. The exercise used one run per harness, task and model, so it supplied no variance estimates.

Composio held DeepSeek V4 Flash constant across eight harnesses

Each workflow operated under a 900-second limit and was checked by a programmatic verifier in isolated fixtures containing decoys and near-identical keys. The workflows covered Airtable, Gmail, Google Calendar, Google Sheets, GitHub, Slack and PostHog. Composio held DeepSeek V4 Flash constant across eight harnesses.

The reported pass rates ranged from 46.7% for OpenCode to 66.7% for Pi Agent. DeepAgents matched Claude Code’s pass rate while costing one quarter as much per successful task. Composio noted that Pi used different reasoning settings across two providers, and Prime Agent generated only 24 gradable runs.

Artificial Analysis averages three attempts across 326 tasks

Artificial Analysis combines DeepSWE, Terminal-Bench v2.1 from the Laude Institute and Scale AI’s SWE-Atlas-QnA. Its index averages pass rates across three attempts on 326 tasks and reports cost per task, token use and wall time. It also compares Claude Code, Cursor CLI and Opencode while holding Claude Opus 4.7 fixed.

In the June DeepSeek run, Codex billed more than one million tokens across the suite, with cache reads accounting for 77% and priced at roughly one tenth of the normal rate. The author reported that Codex then cost less per solved task than Claude Code despite using twice as many raw tokens, and attributed Claude Code’s near-zero cache share to its Anthropic-style OpenRouter endpoint.

Prompt fidelity under load The June test injected 100,000 tokens of irrelevant log noise before each task. Seven configurations transmitted it faithfully; Kilo and Opencode omitted 83%–89% yet reported success. OpenClaw refused to run and Kimi Code crashed, while Claude Code sent the full prompt but performed poorly. The account also notes differing harness-layer pricing approaches at Anthropic, OpenAI, Google and Microsoft, while DeepSeek released swappable runtime components under an MIT license.