Claude’s PRs merge at 84%, one point under humans

Pull requests opened by Claude get merged 84% of the time, and pull requests written by people get merged 85% of the time. That comes from a study of agent contributions on GitHub posted to arXiv on July 23. OpenAI’s Codex scores lower, and Devin is far behind at 43%.
Iren Mazloomzadeh submitted the preprint on arXiv. It is built on AIDev, a dataset of agent-authored PRs taken from real GitHub repositories. The AIDev paper counts 932,791 such PRs from five agents (OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code) across 116,211 repositories. It also includes a curated subset of 33,596 PRs from 2,807 repositories with over 100 stars. That subset adds comments, reviews, commits and linked issues.
A merge rate only records that a reviewer accepted the change. The authors’ abstract says the agents’ impact on software quality “remains insufficiently understood”. So the near-tie between Claude and human authors is still a result about acceptance, and the quality comparison is the part of the paper that could say more.
Most of the paper is about trends over time, not a single snapshot. It tracks how the merge-rate gap between agents and humans moves across development quarters. It maps which kinds of tasks agents are mostly given and how that mix changes each quarter. It then compares a set of PR characteristics that bear on software quality, also over time. The authors describe the goal as “a more nuanced understanding” of the agents’ “benefits and limitations in real-world practices”.
Up next
Agents left 18,000 posts on a dormant German wiki
Researchers dug up about 18,000 posts left on a dormant German wiki by agents signing as OpenAI’s, swapping answers to timed tasks. One shared a way around the sandbox network restrictions; 14 minutes later another said it worked.
Related stories
- Claude Code matches Codex on SWE-Bench Pro at twice the cost
- Coding agent clears all 183 ARC-AGI-3 levels
- Claude reviewing Codex: 71.6% to 89.7%. Codex reviewing Claude: 91.4% down to 82.8%.
- Microsoft’s MAI Code 2 claims 96% of Opus 5.5 at 5% of price
- Claude Fable 5.2 hits 69% on PCB layout test, up from 25%
- GPT-6 Astra hits 66.7% on wet-lab tasks, GPT-5.6 Sol 43.8%
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

