Anthropic paper: automated researchers fix 10/10 benchmarks

Anthropic published a paper on August 28 in which Claude-driven agents ran alignment post-training on a target model on their own. On all 10 benchmarks they were given, they reduced the targeted misaligned behavior without degrading the model’s overall performance. The work, titled “Automated Researchers Can Reliably Mitigate Alignment Failures”, came out of Anthropic’s fellows program and was led by fellow Chen Yueh-Han, TechCrunch reported.
The paper calls the agent an Automated Alignment Researcher, or AAR, and tests it against 28 experienced human researchers. It targets failures that public benchmarks already measure, such as deception, sycophancy and jailbreaks. The code is on GitHub, and the repository lists Chen Yueh-Han, Jiaxin Wen and Jan Hendrik Kirchner as authors.
The loop follows the usual steps of human research, run many times. A Claude agent searches the available literature, proposes a training method and writes the code for it. That code has to be approved by an integrity monitor before anything runs. The target model is then trained with the method for 30 minutes and scored. Methods that raise the score are kept as the starting point for the next round, and the rest are thrown out. Over many iterations the agent hill-climbs the benchmark: it pushes the score up by keeping only the changes that improve it. Claude-haiku models are among the judges that do the grading.
The approval step makes sense in context. Agents checking their own training code haven’t done well: in August, one researcher’s run of a self-distillation task on Opus 4.8 produced six errors in the objective function that its self-check never caught.
Every task in the harness needs at least one benchmark to climb and at least one held-out benchmark that the agent never optimizes against. The held-out one shows whether a gain carries beyond the test it was tuned on. According to the abstract, the strongest AAR methods generalize to the held-out benchmark, to multi-turn behavioral audits and to models up to 4.7x larger than the one they were trained on.
The human baseline is where the paper is most direct. The 28 researchers each had up to eight hours to come up with a one-shot method for the same benchmarks. “The best AAR method beats what experienced humans propose, on average within six hours,” the paper reads, so the agent beat them in less time than the humans were given, and it got many attempts where they got one. The authors also tried giving the AAR human ideas as its starting research direction. “Human guided research directions do not lead to stronger performance,” they write, and they read this as a sign that current AARs may not need guidance from experienced researchers.
The paper also compares costs: “An AAR costs roughly $4 per hour in API inference against the $150 per hour we pay our human researchers.” That makes an hour of the agent about 37 times cheaper than an hour of a researcher. The figure covers inference only, at a per-hour rate.
The paper sets its own limits. The system works only as far as the benchmarks reflect the actual alignment goals. Building and maintaining those benchmarks takes significant work, and so does maintaining and expanding the literature the agents draw on. The conclusion is also scoped to well-characterized failures, the ones that already have a reliable way to measure them.
The engine itself isn’t tied to alignment. Besides the 10-failure benchmark suite, the repository ships a task-agnostic template, generic_aar/, that runs the same loop on any measurable task with a benchmark to climb and one held out. The authors say their results “provide early evidence that automated alignment post-training could become practical in the near term.”
Up next
Leaked Hatch materials point to a Meta agent superapp
Testing Catalog reviewed pre-release materials for Meta’s unreleased Hatch agent: standalone web, iOS and Android builds with browser control and persistent cloud environments. The same files reference Messenger and WhatsApp Companions.
Related stories
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

