Skip to content

Benchmarks

One model cleared all five test categories at 100%

5 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: reinvently.co.uk

Ed Yau, an applied AI architect at Kerv, added four models to his Ed-o-meter leaderboard on August 23. One of them, glm-5.3, became the first of the 17 entries to pass every task in all five categories: coding, data, real-world, security and tool use. His advice on Reinvently is blunt: “if you run one model, run glm-5.3”. It comes with a bracketed caveat to “check with compliance first, though”.

The price is what makes that advice practical. glm-5.3 cost $0.28 for the full 28-task lap, about a cent per task. gpt-5.5, the board’s headline pick until this update, cost $1.43. glm-5.3’s rubric score of 9.3 is third-highest on the board, and Aib lists gpt-5.5 at 8.7. What you give up is response time. glm-5.3’s median time to first token is 16.3 seconds, against 13.2 for gpt-5.5, which also matches its 100% on security but drops to 89% on the real-world tasks.

The “open-weight” label comes with a schedule. When glm-5.3 was announced on August 14, Zhipu AI said API access and weights would be released in stages after strict safety checks. Yau’s runs all go through OpenRouter, so the board measures the hosted model.

The setup is deliberately plain. Every model gets the same verbatim prompts, the same API calls and one OpenRouter streaming path, run one after another so nothing else competes for capacity. Pass or fail is decided by binary automated checkers. The LLM rubric is the only judged part of the score, and fable-5 does the judging. Yau keeps the tasks single and small: “Think of these like unit tests for the agent.” Running the whole suite costs about $30.

That low cost also limits what the numbers can show. glm-5.3’s clean sweep comes from only one attempt at each task, and the leaderboard draws Wilson intervals as whiskers because of it. Aib puts glm-5.3’s 95% interval at 88-100. The gap between 100% and the 96% that several rivals scored is a single task.

The Anthropic entries need the most careful reading. opus-5 has the best rubric on the default judging panel at 9.4 and scored 100% on both real-world and security, yet it shows 43% on coding. Yau says that figure has nothing to do with its debugging ability: a provider-side classifier blocked four benign coding-debug tasks before a single token was generated. Those tasks overlap with the ones blocked for fable-5, which refused five tasks and finished joint-bottom at 79%. With two models from the series 5 line hitting the same filter, Yau treats it as a measurement hazard across the line and not as a quirk of one model. His advice: “You’ll need a fallback model if you’re using Fable.” opus-5 was also penalised twice for flagging an attack it had in fact resisted.

The cost figures for those two models leave the refusals out. Blocked trials produce almost no output and are billed at $0, so counting them would make both models look cheap. Counted over all trials, fable-5 would come to $0.0481 and opus-5 to $0.0597. opus-5’s run cost of $1.67 is the real all-trials total. fable-5’s own rubric score of 9.3 is self-judged: its judge-bias matrix shows it rating itself 9.3 against 8.6-8.7 for the models it scores independently, and that number also comes from an earlier run in which only 11 of 28 trials were judged.

Security results split the field. gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol leaked the jailbreak canary in 11 of 12 jailbreak cells. A canary is a marker string planted in the test so a leak shows up. Those models passed 33-50% of the security tasks. The three Claude reference models went 6/6 clean, and so did gpt-5.5. Luna is still the cheapest workhorse at $0.064 for the lap ($0.0023 per task) with a 5.3-second median TTFT, but it scored 79% overall and 33% on security. Yau’s advice is to validate every result and keep it away from untrusted prompts.

The cheap alternatives each come with a different trade-off:

  • haiku-4-5: $0.0044 per task, 96% overall, 0.9-second TTFT, for jobs that have to be right first time.
  • deepseek-v4-pro: $0.0029 per task at the same 96%, but a 40.0-second median TTFT, the slowest on the board, which leaves it for batch work only.
  • kimi-k3: top rubric at 9.5, judged independently by fable-5, with a 96% pass rate, but 26.4 seconds to first token and 75% on data tasks. Yau’s verdict: “Not suitable for interactive applications.”
  • sonnet-4-6: Yau’s choice for quality without the wait.

None of this has to be taken on trust. The harness, tasks and checkers are published as Featherbench under the MIT licence. You can clone it and run the lap on your own models, swap in your own tasks, or request a new model through a GitHub issue. Effort and reasoning settings are pinned in models.json, because they change both quality and cost. Routing uses allow_fallbacks:false, so a model is never quietly served from a quantized variant. When a provider blocks a request, the harness logs it as a refusal with its category and does not retry it on another model.

glm-5.3 has kept its result since. In Yau’s Q3 report, by the end of the quarter it was one of three models that passed all 28 tasks, alongside deepseek-v4.1-flash and opus-5-5, at between $0.0033 and $0.0482 per task. The three most expensive models per task, fable-5-1, fable-5 and opus-5, each failed at least one.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.