Devin's PRs merge at 43%, Claude's at 84%

AI coding agents differ sharply in how often their code is accepted: pull requests from Claude merge at 84%, from Codex at 74% and from Devin at 43%, against 85% for human contributors. The figures come from a preprint posted to arXiv that draws on the AIDev dataset.

At a glance

• Only Claude, of the three agents measured, comes within a percentage point of the 85% human baseline; Codex trails that baseline by 11 points and Devin by 42 points.
• The work draws on the AIDev dataset and tracks how the merge-rate gap between agentic and human pull requests has moved over time rather than fixing it at a single point.
• Alongside merge rates, the authors classify which development tasks agents are pointed at, how that task mix shifts by quarter, and how agentic pull request characteristics compare with human ones.

The spread matters more than the headline parity. Claude sitting a point below the human baseline and Devin at roughly half of it suggests that the useful unit of comparison is the individual agent, not agentic contribution as a category, and that a team's review burden likely depends on which tool it standardises on. Merge rate is also a crude proxy: it records what maintainers accepted, not what held up afterwards.

A merged pull request is one a project maintainer accepted into the codebase, and the rate at which that happens is the measure the study builds on. The authors track it longitudinally, examining how the distance between agentic and human pull requests has changed over time rather than reporting a single snapshot.

The second strand identifies which development tasks agents are predominantly applied to and how those task distributions evolve across development quarters. The third compares key characteristics of agentic and human pull requests, with the authors framing the comparison around implications for software quality and the temporal dynamics of both groups.

The paper's stated premise is that the impact of coding agents on software quality remains insufficiently understood, and that how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. It is filed under Software Engineering with a cross-listing to Machine Learning.

What v1 does not settle

The preprint went up on 23 July 2026 as version one, submitted by Iren Mazloomzadeh and indexed as arXiv:2607.21832, released outside a peer-reviewed venue. The abstract does not give the observation window, the number of repositories behind the merge rates, or the quarter-by-quarter task figures, so the breakdown that would account for the 41-point spread between Claude and Devin sits in the full text rather than the summary.