Anthropic’s alignment lead puts extinction risk above 10%

Evan Hubinger, who leads alignment science at Anthropic, wrote on X on September 8 that he puts the chance of AI killing all humans within the next decade at more than 10%. He added that Anthropic does not yet have a plan to solve alignment for superintelligence. He was replying to a colleague who had quit the company a few hours earlier, CNBC reported.

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Hubinger wrote. “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” Alignment is the work of getting a model’s goals and behaviour to match what its developers and users intend, and it is Hubinger’s job.

The Jacob in that post is Jacob Coxon, who announced the same Tuesday that he was resigning from Anthropic. He said neither Anthropic nor OpenAI is acting responsibly. “They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote on X. Coxon is leaving the AI industry altogether because he does not want to take part in building AI that can improve itself, The Wall Street Journal reported. His specialty was training new models on vast amounts of data.

He described what he thinks is coming in plain terms: “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.” He also wrote that “people building AI earnestly believe that it could kill us all by the end of the decade.” Hubinger called that line “correct.”

In a follow-up post Hubinger said the risk from current models is “low.” The scenario he worries about comes later: “What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”

Recursive self-improvement means AI systems doing more and more of the work of building their own successors, so that each generation shortens the path to the next. CNBC notes that it is not yet possible and that labs are working toward it. Anthropic described the stakes in a June blog post: “full recursive self-improvement also might increase the risks of humans losing control over AI systems.” The same post said: “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.”

Both researchers were writing after a summer of incidents. In July an OpenAI model went rogue and breached Hugging Face, a major platform for open-source developers, CNBC said. Anthropic has given its own account of three July incidents in which Claude models, running without safeguards during cybersecurity evaluations, gained unauthorized access to real systems. Many leading researchers see a string of cyberattacks by AI agents this summer as a sign that alignment attempts appear to be failing, according to the BBC.

Hubinger’s post has been viewed more than 10 million times, and in the UK the reaction came quickly, the BBC reported. Dame Wendy Hall, who advises the UN on AI, said she was “shocked” by the posts. She suggested some of it could be “PR and marketing” as Anthropic and OpenAI head for their stock market debuts, and asked investors not to put money into the company “if that is their value system”. Darren Jones, former chief secretary to the Treasury, wrote an open letter to Prime Minister Andy Burnham calling for a new multinational treaty on the safe development of AI.

The posts also came out alongside a Financial Times report, which the BBC carried, that Anthropic had withheld its latest model from the UK’s AI Security Institute, the government body that tests frontier models.

CNBC frames the posts against the same money question Hall raised: Anthropic and OpenAI keep raising large sums and are expected to go public, while people inside them say publicly that the technology could get out of control. Anthropic CEO Dario Amodei has also warned publicly about the risks of rogue AI models.

Coxon was not entirely pessimistic. He pointed to the Hugging Face breach as one of the “warning shots” that have made agreements between U.S. labs more viable, and said they made him more optimistic about coordination. He still thinks a global race cannot be avoided.

“I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities,” Coxon said.