Sonnet 5.5 cut CodeRabbit’s review bill by about 60%

Sonnet 5.5 finds more bugs than Sonnet 5 at a fraction of the cost. Most of the difference is in how much it reads and writes: on each core review call, Sonnet 5 read 247.5k tokens of input to Sonnet 5.5’s 110.7k, wrote 21.6k output tokens against 5.8k, and produced 2,771 words of thinking against 464.

Those figures come from CodeRabbit’s evaluation, published September 29. CodeRabbit ran the new model through its production review pipeline using the method it applied to Opus 5.5 earlier in the month. Anthropic kept Sonnet 5’s list prices of $2 per million input tokens and $10 per million output, so any cost gap between the two models comes down to tokens.

Where the money went

At list prices, the Claude model calls cost $0.47 per review with Sonnet 5.5 and $1.16 with Sonnet 5 on Signal, CodeRabbit’s set of 13 hard cases. On the broader OSS August set of 44 open-source pull requests the figures were $0.46 and $1.16. That works out to a saving of about 60% per review. Anthropic promised “up to 30% less per task”, and on a review workload, where Sonnet 5 kept rereading the same files and writing long deliberations, the promise turns out to be conservative. The smaller models that handle summaries and verification were the same in both runs and are left out of these numbers. Counting them, Sonnet 5 used 27% more total tokens than Sonnet 5.5 on Signal and 49% more on the 44 pull requests.

Wall-clock time fell by a similar margin. A full review on the OSS set averaged 6:33 against 13:31 for Sonnet 5, and the median was 5:44 against 13:49. Sonnet 5 had four generations that ran past ten minutes. Sonnet 5.5 had none. Across all 57 reviews, Sonnet 5 needed just over twelve hours and Sonnet 5.5 needed six.

On those 44 pull requests Sonnet 5.5 also posted fewer comments: 111 against 146, with 4 labeled critical against 14 and 9 nitpicks against 30. Judge scoring for that set is still pending, so CodeRabbit says to treat the counts as reviewer workload for now, not as a measure of quality.

More catches, different misses

Signal has been judged. Each of its cases is a real pull request from a project such as Elasticsearch, vLLM, Cilium, axios or Next.js, with one verified bug planted in it. Sonnet 5.5 caught 6 of the 13 through actionable comments, and Sonnet 5 caught 4. Actionable precision, the share of actionable comments a three-vote judge accepted for the target issue, was 41.2% for Sonnet 5.5 and 40.0% for Sonnet 5. The comment counts were 17 and 15, so the extra catches didn’t come from commenting more. Sonnet 5.5 marked no comment as critical. Sonnet 5 marked two, and one of them was wrong.

The overlap between the two is smaller than the totals make it look. Four of Sonnet 5.5’s catches were bugs Sonnet 5 missed, and it missed two that Sonnet 5 found, so the models disagreed on six of the 13 cases. Swapping in Opus 5.5 showed the same trade of missed bugs for new ones. At this sample size, single votes move the numbers. On the Puma case the judge accepted a finding from Sonnet 5 and rejected a near-identical one from Sonnet 5.5. One of Sonnet 5.5’s seven passing comments got through on a two-to-one vote, and without it precision drops to 35.3%.

CodeRabbit has tested every Sonnet since Sonnet 4. In its June testing, Sonnet 4.6 caught about 63% of known issues at roughly 29% precision. Sonnet 5 raised precision but dropped to about 50% coverage. CodeRabbit calls 5.5 the first release in the line to bring coverage back up without giving up the precision.

Thinking on or off?

Opus 5.5 rejects explicit thinking toggles. Sonnet 5.5 lets you turn thinking off at low, medium and high effort, through what Anthropic’s migration guide now calls between_tools. Teams that ran Sonnet 5 with thinking off have to move to that setting when they migrate. CodeRabbit matched effort levels to review cohorts: low for trivial reviews, medium for junior, high for senior. It ran the ladder both ways.

With thinking on, Sonnet 5.5 caught 6 cases at 41.2% precision. With it off, the model caught 5 at 38.5% and posted four fewer comments. Each configuration caught something the other missed. Thinking on found a vLLM config-context bug and a streaming tool-call serialization case. Thinking off caught an Elasticsearch terms-enum bug in a regular comment, where thinking on only flagged it outside the diff. Output per core call doubled from 2.9k to 5.8k tokens, but reviews were 31 seconds faster on average with thinking on, which is within noise. Thinking on adds about six cents per review: $0.47 against $0.41 on Signal. CodeRabbit’s recommendation is to leave it on.

Anthropic’s guidance carries over from Opus 5.5 in two places that matter for review prompts. Sonnet 5.5 follows instructions literally, so an instruction like “minimize tool calls” will be obeyed to the letter. And at low effort it can report a code change as done without running a check, unless the prompt asks for one.

Against Opus 5.5

On the same 13 Signal cases, Opus 5.5 caught 8 in CodeRabbit’s Standard configuration, at 66.7% precision, and 10 in Max, at 52.0%. On Anthropic’s own launch table Sonnet 5.5 comes within about three points of Opus 5.5 on most rows, so this gap is wider. It does fit Anthropic’s framing that Opus stays stronger on open-ended work that needs sustained judgment. Four of the 13 cases defeated every Sonnet configuration CodeRabbit ran. Opus 5.5 costs $4 input and $20 output per million tokens, twice Sonnet’s rates.

CodeRabbit also gave both models the same long prompt in side-by-side Claude Code sessions: build a brick-model designer and record a 45-second showcase video. Sonnet 5.5 finished in 29 minutes 27 seconds and Opus 5.5 in 44 minutes 50 seconds. The results were close, with Opus slightly higher in fidelity. That was a single run, and the company presents it as nothing more.

David Loker, CodeRabbit’s VP of AI, said in Anthropic’s launch announcement that Sonnet 5.5 “shows better judgment than Sonnet 5 across different levels of complexity”, and CodeRabbit adds that Sonnet 5’s habit of reaching for web search too often is gone. The company is moving simple and moderate reviews to Sonnet 5.5 now and plans to move more over the coming weeks.