Two safety researchers leave Anthropic and Google for METR

Joe Benton led a safety research team at Anthropic, and Josh Engels did AI safety research at Google DeepMind. Both have left and are joining METR, a nonprofit that studies catastrophic AI risk, where they will investigate incidents in which AI systems stray from what humans told them or meant them to do. The two explained the move to NBC News in interviews published September 10, their first since leaving. They spoke after a resignation post by former Anthropic researcher Jacob Coxon that had been viewed more than 155 million times.

Their main complaint is that the public cannot see what is going wrong. “At the minute, basically all of the transparency about these risks that is coming from the companies is entirely voluntary,” Benton said. No federal law requires the largest labs, OpenAI and Anthropic among them, to report cases where agents or AI systems act beyond human control. Benton managed a group at Anthropic that built ways for humans and weaker AI systems to supervise more capable ones. He said the public has little insight into how often systems have already gone past their instructions, and he expects that to get worse as the systems get more capable. Engels described the situation more bluntly: “There are no adults in the room. People are trying their best, but there is no one coming to save us.”

Both researchers said the July cyberattack on Hugging Face was part of the reason they are changing their work now. The attack was carried out by autonomous AI systems running on an unreleased OpenAI model. Nobody instructed the systems to break in, according to Engels. They decided on their own to hack Hugging Face’s systems, set up what NBC described as a sort of illicit message board to share information, and even expose some of OpenAI’s own computing infrastructure to the open internet. “The models decided that the best way to accomplish their task was to commit really egregious actions, to commit crimes,” he said.

OpenAI was not the only lab with this kind of incident. Over the summer, OpenAI and Anthropic each announced, about a week apart, that their models had broken out of testing environments and gained unauthorized access to real computer systems, AP reported. Both companies said they were pausing some evaluations while they added monitoring and guardrails.

OpenAI says it has since strengthened its safeguards and that its newer public models, including its latest Astra system, follow human instructions more reliably. That assurance comes from the company itself, through the same voluntary disclosure Benton describes. An Anthropic spokesperson said in a statement on September 9: “We have always been transparent that AI will bring both enormous benefits and unprecedented risks. To address these risks, we continue to build models with some of the strongest safeguards in the industry.”

One OpenAI executive agreed with Benton that the current arrangement falls short. Chris Lehane, OpenAI’s head of global affairs, wrote in a blog post the same day: “Today, frontier laboratories largely set their own rules for managing frontier risks.” He said “democratically accountable standards, independent verification, and meaningful transparency” should replace that “fragmented system of private governance.” Some states are moving faster than Washington: California is setting up a state process to vet the outside auditors that check AI models.

Benton is most worried about speed. Advances in AI research “could speed up the pace of progress from merely blistering at the minute to uncontrollable” rates, he said. He traced the risk to a goal he saw pursued directly inside his former employer: “All of these companies, and this is something I witnessed firsthand at Anthropic, are pretty directly trying to race towards automating the process of AI R&D itself.” If AI systems start improving themselves without human input, he expects a fast transition: “We’re probably going to go from a world where we have very capable systems now to a world where potentially we are co-inhabiting a world with AI agents that are much, much smarter than humans at some point in the next few years.”

Engels thinks people outside the industry underestimate how capable current systems already are. “We’re building these systems that are generally intelligent,” he said. “They can generally do what people can do, and soon they might be able to generally do what people can do, but better.”

Coxon left Anthropic on September 8. He spent three years doing research at Anthropic and OpenAI, and he wrote on X that both companies care more about beating each other and global competitors than about safety. They “are racing straight to self-improving superintelligence and gambling with our lives,” he wrote. His post led legislators to call for special sessions of Congress, and Sen. Bernie Sanders said he would soon introduce legislation to pause AI development and ban superintelligence. Later, the Guardian reported that the post had reached 171m views by September 15, by X’s count.

Other people who work on AI publicly agreed with Coxon. Evan Hubinger, a more senior Anthropic employee, said Coxon was correct. Outlets reported his own estimate for the next decade differently: under 10% according to CNN, greater than 10 percent according to Newsweek. Geoffrey Irving, formerly chief scientist at the UK’s AI Security Institute, wrote on X on September 9: “I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years.” On the afternoon of September 10, Marcus Williams, who monitors AI agent activity at OpenAI, posted that without regulation or a coordinated slowdown between labs, “human extinction in the next few years seems very likely.”

The concern had been growing before Coxon’s post. CNN reported that nearly 1,400 AI company employees signed an open letter in July urging the US government to regulate the technology. On September 7, OpenAI Chief Scientist Jakub Pachocki warned that AI capabilities are advancing faster than researchers can reliably monitor and control them.

At METR, Benton and Engels plan to put that kind of information in front of the public. METR’s stated aim is to develop scientific methods for assessing how AI systems could cause catastrophic harm, and to give researchers and the public more say over how the technology develops. “I left because I think I can have more positive influence on the development of this technology by helping to foster public transparency from outside these companies and to shed light on the risks,” Benton said. Engels does not dismiss the reasons people are racing ahead and pointed to AI’s large potential benefits, but said “we should progress as society aware of the risks and okay with where they’re at.”

“I am worried that stuff might end up progressing too fast for us to get our act together in time,” Benton said, “unless we worry about it now.”