Opus 5.5 finds new bugs for CodeRabbit and misses old ones

CodeRabbit ran Claude Opus 5.5 through its code-review pipeline and got the opposite trade from Opus 5. On a broad open-source test, coverage rose a little and precision slipped a little. On a small set of harder cases, both rose sharply. The details are in a September 22 post on the CodeRabbit blog, and the catch is how much extra work the model does to get there.

CodeRabbit tested two setups against its production model mix and named them Standard and Max. Standard uses lower reasoning-effort settings across the pipeline and Max uses higher ones. Neither is a single API effort value or Anthropic’s default configuration. The main test, OSS August, covers 80 known bug patterns from open-source code. A separate set called Signal holds 13 harder cases. Every comment was counted after verification, deduplication and filtering. Precision here is the share of comments that the benchmark judge accepts for the target issue. It does not measure whether developers accepted them.

The Opus 5 result is what makes this one worth reading. In the Opus 5 write-up, the x-high configuration raised actionable precision to 39.3% from the baseline’s 35.2%. But it caught fewer known issues, 55.2% against 61.1%, and produced roughly four times as many nitpicks. CodeRabbit says the two evaluations used different test sets and configurations, so the comparison is historical context and not a measured improvement between versions.

On OSS August, the production baseline caught 49 of 80 issues (61.3%) at 39.3% actionable precision, with 116 reported comments. Standard caught 51 (63.8%) at 38.6%, with 127 comments. Max caught 50 (62.5%) at 35.7%, with 140 comments. Against the baseline, Max left 24 more comments for one extra catch, so on the broad test the higher effort mostly bought more reading. CodeRabbit’s summary is that “turning up effort did not consistently produce a better review.” Max also tagged a larger share of its comments as “minor”. CodeRabbit points out that the tag doesn’t make a finding invalid.

The headline totals hide churn. Standard caught 11 open-source issues the baseline missed and missed nine that the baseline caught, which works out to a net gain of two. Switching reviewers changes which bugs get through even when the score improves. The example CodeRabbit chose is a race in Cal.com. Two overlapping jobs both read retryCount = 0 and both write 1, so two increments leave the counter at 1. The fix sends retryCount: { increment: 1 }, which has the database increment its current value, so both increments are kept. Both Opus 5.5 setups flagged the race and proposed the atomic increment. The production baseline missed it.

Signal is where Max makes its case. Through regular actionable comments, Max caught 10 of 13 issues (76.9%) at 52.0% precision with 25 comments. Standard caught 8 (61.5%) at 66.7% with 21 comments, and the baseline caught 5 (38.5%) at 29.4% with 17. Findings outside the changed lines can still be valid catches, and once CodeRabbit counted them, Standard and Max each reached 10 against the baseline’s 7. They didn’t miss the same three, though. The major and minor counts in that table are severity labels the model assigned to its comments. They aren’t counts of distinct bugs.

Price is the other half of the story. Opus 5.5 lists at $4 per 1 million input tokens and $20 per 1 million output tokens, down from $5 and $25 for Opus 5, a 20% cut. Cache reads drop 60%, from $0.50 to $0.20 per 1 million tokens. Against that, every tested configuration used more tokens than CodeRabbit’s production mix. Standard used 49.2% more on OSS August and 40.6% more on Signal. Max used 57.6% and 60.1% more. These figures come from the evaluation team’s run summary. They don’t split input from output and don’t convert into dollars or latency. They also measure something different from the price cut, which is against Opus 5, while the usage figures are against CodeRabbit’s own mix. Anthropic’s September launch pitch put Opus 5.5 at 40% lower running cost than Opus 5 at default settings. CodeRabbit’s runs are not a test of that claim either way.

A few API changes matter to anyone running a review pipeline. Thinking is always adaptive: Opus 5.5 rejects requests that explicitly turn thinking on or off, so effort becomes the main control over deliberation, latency and cost. The recommendation is to start at medium and test low and high alongside it. At the same effort setting, Opus 5.5 can think more per turn than Opus 5, especially at x-high and max. Reportedly, Opus 5.5 at medium effort matched or beat Opus 5 at high effort on multi-step coding tasks with roughly half the tokens. Forced tool calls are retired, so an application can no longer require the next response to call a particular tool. Schema enforcement still constrains a tool call’s arguments, but it doesn’t guarantee that the call happens, so workflows have to check for the call and handle the case where it’s missing. Fast mode is available, the model doesn’t require data retention, and a biology classifier is new relative to Opus 5. CodeRabbit’s benchmarks don’t measure any of these.

For teams running their own evaluation, CodeRabbit suggests listing the bugs only the current reviewer catches next to those only the replacement catches, and weighing their impact on the application. On cost, it suggests tracking input, output, cache-read and cache-write tokens separately across a full review, including retries, verification calls and fallback models. A short visible answer can still use a lot of tokens, because thinking draws on the same budget. The differing catches make a case for testing a second reviewer next to the first. CodeRabbit notes that these runs say nothing about the quality, cost or comment volume of running both together.

The post also covers a coding side project. Gowtham, who worked on the evaluations, gave Opus 5.5 a goal for a personal project and let it run overnight. The team also had it build a game inspired by Grand Theft Auto: San Andreas, which took longer than the version Fable 5.1 built. Then Opus 5.5 wrote bots that read live game state and drove all three games with keyboard and mouse input. CodeRabbit keeps those impressions separate from the review numbers, and its verdict ends on cost: “Its efficiency remains an open question, so teams should verify whether its lower token prices actually translate into lower production-review costs.”