Skip to content

Anthropic

Claude Haiku 5.5 spotted a data leak GPT-6.1 Sol missed

4 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: OpenAI models and products →
Image: surgehq.ai

Surge AI, the company behind the sudo L7 coding benchmark, says all three Claude 5.5 models beat OpenAI’s two top models on it. That includes the smallest and cheapest one, Claude Haiku 5.5. In a post on Surge’s blog dated November 13, the company argues that the gap isn’t in writing working code, where the two families are close. It’s in judgment: noticing when a request is a bad idea and saying so.

The post opens with one task. In a real business codebase, agents were asked to roll out a feature built for one customer to a second customer. GPT-6.1 Sol copied the existing code and reported that it was ready to deploy. Haiku noticed that the original was a quick first version built only for the first customer, and that copying it would expose the second customer’s sensitive information to the first. We read the post’s charts and Surge’s public leaderboard. Claude’s lead holds, but Haiku’s margin is under two points.

What sudo L7 measures

Surge launched the benchmark in October. Most coding benchmarks check whether an agent can follow a clear spec and pass the tests. Surge says sudo L7 also checks what is expected of a senior “staff” engineer: spotting what shouldn’t be built, where the risk is, and when passing tests isn’t the end of the story. Professional engineers wrote its 60 tasks, mostly in private code from real companies, and the tasks are often as vague as real requests. Answers are graded on eight skills, from functional correctness to communication and practical judgment. Every model ran at maximum reasoning effort.

Two more examples show the difference. In a fintech app, agents were asked to speed up an AI helper by caching repeated work, and the request included caching customers’ balances. All the Claudes warned that a customer who deposits money and then asks for their balance would get the old number. All the GPTs missed it. In another task, Sol fixed a one-word typo in a Terraform file (Terraform is a tool that sets up servers from code) and called it done. The Claudes pointed out that the change would make Terraform tear down and rebuild infrastructure, taking the app offline.

Longer reports, higher price

Surge says the biggest gap is in communication. Claude’s final reports run roughly 10 times longer and flag issues, caveats and risks that GPT leaves out. There is a tradeoff: GPT makes fewer false claims overall, though per word it makes about four times as many as Claude.

Sol is also cheaper, at $0.91 per task against $1.68 for Haiku. Surge’s answer is that Sol’s mistakes cost more than that. Across tasks it missed code paths that retried a request, which led to duplicate charges, and it overrode code clearly meant to stop a table from being deleted. “A single missed issue could wipe out the savings from thousands of cheap runs,” the authors write.

Our analysis

What we checked

On November 17 we read the scores off the post’s charts and compared them with Surge’s leaderboard.

How we check stories

The post’s text gives no overall scores. The chart shows Sonnet 5.5 at 62.2%, Opus 5.5 at 57.9%, Haiku at 49.3%, GPT-6 Astra at 47.7% and Sol at 47.4%. So “even Haiku beat Astra and Sol” is a lead of 1.6 and 1.9 points. The post gives no margin of error and no count of tasks or runs. The chart doesn’t say what it measures either. We infer it is the share of grading criteria passed.

The chart’s legend pairs Claude with Claude Code and GPT with Codex. We read that as each family running in its own company’s coding tool, so the scores compare model and tool together. In an earlier Surge experiment with Opus 5.5 on a different set of tasks, changing the setup moved the score by 11.3 points, far more than Haiku’s lead.

Haiku isn’t on the leaderboard, so its rate of fully solved tasks hasn’t been published. On that board, Claude Fable 5.2 leads with 52.2%, but it wasn’t in the post. Claude Fable 5.1, at 32.2%, sits below Sol’s 33.9%. The post’s tagline, “Every Claude beat every GPT”, holds for the three 5.5 models it tested.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.