Copilot wins GitHub’s benchmark, ranks fourth on Martian’s

GitHub Copilot code review tops the first leaderboard of ReviewBench, an open benchmark for AI code reviewers that GitHub developed with Microsoft and unveiled on October 5, The New Stack reports. If your team is shortlisting a reviewer from that table, every score on it so far comes from GitHub’s own runs: by GitHub’s account, its team tested the public version of each product, and the vendors neither conducted nor verified those tests. Martian’s independent Code Review Bench puts Copilot fourth on its online board.
Copilot was tested on October 1 in its Balanced configuration, the more thorough review mode GitHub made the default on September 28, so the board scores the setting Copilot has used out of the box since then. On private repositories, those reviews are also billed in GitHub Actions minutes. Cubic and Greptile were tested back in June, and ReviewBench itself cautions that the products may have changed since and that results on its corpus may not translate to a company’s own code.
Copilot’s 40.1%, scored in the Balanced configuration, is a grounded F1: it combines precision and recall against the findings already in ReviewBench’s reference set. GitHub’s announcement post also defines augmented metrics, where the judge rules on findings that match nothing in that set, so a reviewer can earn credit for a valid issue that no source of the set surfaced. Because augmented recall expands its denominator according to what each agent discovers, GitHub keeps grounded recall as the headline comparison across systems. The leaderboard can be sliced by severity and category, and changing β in its Fβ score shifts weight toward recall or toward precision, re-ranking the table.
What counts as a finding?
ReviewBench takes 219 public pull requests from 187 public repositories in 19 programming languages and has every reviewer inspect the same changes. GitHub chose the corpus after analyzing 103.9 million pull requests: its language and repository-size distributions closely mirror GitHub overall, while pull request size is deliberately weighted away from tiny, single-file changes.
What a reviewer ought to find comes from a reference set pooled from sources that include human review comments, later changes by the pull requests’ authors, static-analysis tools and LLM reviewers. Findings that describe the same underlying issue are deduplicated, so agreement between sources does not inflate the set, and a finding counts as a true positive only if it is true, relevant and non-trivial, whatever its source. Claude Sonnet 5 classifies findings, and a separate LLM matcher decides whether a reviewer’s finding corresponds to an issue in the set.
The methodology says 47 findings first classified as true positives were corrected by hand, and that human and classifier judgments agreed 96.6% of the time on whether a finding was a true or false positive. Before release, senior engineers who had not helped build the dataset re-labeled every ground-truth finding from scratch. GitHub publishes the dataset, methodology and judging setup, versions the dataset, judge and matcher behind every evaluation, and calls the current release a research preview.
Martian’s two boards
Martian, an AI research company, launched its Code Review Bench in February. It pairs an offline test with an online tracker built on how developers respond to review comments across real open-source repositories. As of October 6, the online board has Cubic first at 64.9% F1, followed by Greptile and CodeRabbit, with Copilot only fourth at 60.9%. On the offline board, Qodo Deep leads, followed by Cubic and Augment, and Copilot sits fifth with a 58% F2 score, a variant that gives recall more weight than precision and so rewards catching a larger share of the relevant issues.
None of these scores compares directly with Copilot’s 40.1%: ReviewBench’s default ranking also uses an F1-style measure, but on a different dataset and scoring method, and Martian’s two boards measure different kinds of evidence. At launch, Martian said its offline results diverged from what it observed in real-world use, and it made the online board its headline metric. Alejandro Carderera de Diego, a staff applied engineer at GitHub, wrote on LinkedIn the day ReviewBench launched that GitHub has been using it to improve Copilot code review and that its “offline results have consistently anticipated the direction of later production experiments.”
LangChain published a different ReviewBench in July: a smaller, model-focused evaluation built from 59 tasks drawn from its LangSmith codebase, with no public vendor leaderboard. LangChain ran different models through the same Deep Agents setup, so the benchmark measures model performance under one common agent configuration.
Martian has made independence a core part of its pitch. Its launch post argued that keeping a credible benchmark takes considerable money and effort, particularly when both the tools under test and the evidence used to judge them keep changing, and that this has historically pushed benchmark development toward academia or the vendors themselves. “We’re trying a third option: a well-funded research lab that doesn’t train models or sell coding tools, and has no stake in which tool wins,” its authors wrote. The benchmark and its methodology are open source under an MIT license, and Martian has invited tool builders, model makers and researchers to contribute.
On ReviewBench, vendors can now submit their own runs through a self-service system, and new results join the leaderboard as they are scored. Each vendor chooses which configuration to put forward.
Up next
OpenAI trains GPT-6 Astra on real Ironclad contract work
A legal ops team wants Finance to approve big purchases and Legal to check odd terms. GPT-6 Astra builds flows like that in Ironclad in an estimated 19.2 minutes per attempt on average, compared with 37.0 for GPT-5.6 Sol.
Related stories
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

