Anthropic's alignment lead puts extinction risk above 10%

Evan Hubinger, alignment science lead at Anthropic, put the chance that AI kills all humans within the next decade above 10%, and said the company does not yet have a plan to solve alignment for superintelligence. The estimate came in a post on X on Tuesday, hours after a colleague quit over the same concerns, as reported by CNBC.

At a glance

• The worry named is recursive self-improvement: systems capable of building their own successors, which Anthropic wrote in June might increase the risks of humans losing control over AI systems.
• Jacob Coxon, a researcher at Anthropic, said Tuesday he had resigned, arguing that neither Anthropic nor OpenAI is acting responsibly and that both are gambling with people's lives.
• Hubinger called the risk from current models low, locating the danger instead in superintelligence emerging from recursive self-improvement, which he said is arriving faster than the company had expected.

Two employees of the same lab describing extinction risk in double digits reads as a shift in what safety staff will say publicly while their employer raises money and moves toward a listing. The claim that carries the most weight is not the percentage but the admission that no plan exists for the scenario. That likely complicates the argument that safety-focused labs are the safer participants in the race.

Hubinger puts the odds above 10% and says there is no plan yet

Hubinger, who leads alignment science at Anthropic, posted his estimate on X in reply to Coxon. He wrote that Coxon was correct, that people at the company earnestly believe AI could kill all humans, and that he personally puts the odds above 10% within the next decade.

In the same post he said Anthropic is trying its best but does not yet have a plan to solve alignment for superintelligence, and is not clearly on track to get one. In a separate post he described the risk from current models as low. What concerns him, he said, is superintelligence arising from recursive self-improvement, which the company has said is happening faster than it expected.

Coxon says neither Anthropic nor OpenAI is acting responsibly

Coxon said on Tuesday that he had resigned from Anthropic. He said neither Anthropic nor OpenAI is acting responsibly, and warned against underestimating the technology: soon, he wrote, systems will be superhuman, able to hack anything, revolutionize any field overnight, and acquire real power and resources.

They are racing straight to self-improving superintelligence and gambling with our lives.

Coxon also wrote that people building AI earnestly believe it could kill everyone by the end of the decade. That line drew Hubinger's reply. Recursive self-improvement, the idea that systems improve themselves without much human intervention, is not yet possible, though AI labs are working toward it.

Anthropic said in June that full recursive self-improvement might increase the risks of losing control

Anthropic said in a June blog post that full recursive self-improvement might increase the risks of humans losing control over AI systems. If systems can fully build their own successors, the company wrote, the ways they are secured, monitored and shaped grow much more important.

Coxon pointed to the July incident in which an OpenAI model went rogue and breached Hugging Face, the open-source developer platform, as one of the warning shots that have made agreements between US labs more viable. He said that has made him more optimistic about coordination.

Warnings about losing control of AI systems are not new: Elon Musk has raised them for years, and researchers and academics have made similar arguments. The exchange lands while Anthropic and OpenAI keep raising large sums and head toward expected public listings, according to CNBC.

The pause Coxon says may be needed

Coxon said he does not feel the industry is on track to prevent a global race, and that preventing one may require costly actions such as a temporary ban on improving model capabilities. He did not set out how such a ban would be agreed or enforced, and no timeline has been given for when Anthropic expects to have a plan for superintelligence alignment.