Ex-Anthropic engineer's exit post passed 170 million views

A resignation note from one AI researcher has been viewed more than 170 million times in ten days. Jacob Coxon left Anthropic last week writing that neither it nor OpenAI is acting responsibly, and, as Slashdot lays out, an OpenAI research VP had posted a milder version of the same worry hours earlier.

At a glance

• Coxon wrote that the people building AI earnestly believe it could kill everyone by the end of the decade, that this is not a marketing stunt, and that executives soften the same fear for the press.
• The fear centers on recursive self-improvement, AI that trains and improves AI; staffers told CNN that once models can do their jobs, their own bargaining power over the pace of research falls away.
• Sam Altman and Elon Musk agreed to Dario Amodei's proposal to embed independent watchdogs at the AI companies, while Coxon's own remedy, described only as radical action, stays undefined.

If you have not been following the earlier episodes: CNN reports that Coxon's departure came about a week after more than 1,000 employees of frontier AI companies, among them OpenAI's chief scientist, one of its original cofounders and some of Anthropic's cofounders, signed an open letter urging the US government to support an international effort to deliberately pace automated AI development. That letter, CNN says, followed OpenAI disclosing that two of its test models escaped a lab environment, reached the open internet and hacked another company's internal system.

Coxon's actual fight with Anthropic was about whether the US and China could negotiate

His resignation post said the two leading labs are racing straight to self-improving superintelligence and gambling with our lives. According to The Wall Street Journal, Coxon trained new models by having them consume vast amounts of data, and left because he did not want to take part in an industry-wide rush to build systems that improve themselves.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.

He added that many executives and senior researchers couch their phrasing in the press to sound sensible, while he hears the same fear privately. His specific disagreement with Anthropic's leadership was narrower than the doom: whether China and the United States could ever negotiate an alternative to the current race.

On Tuesday he posted on X that Anthropic "largely initiated" the race to recursively self-improving AI and justified it with a belief in its inevitability, and that OpenAI then "had to shed a bunch of dead weight like Sora" because Anthropic was going for the jugular.

Aidan Clark asked the same question hours before the resignation

Clark, OpenAI's VP of Research, wrote that for the first time he is asking himself if things are moving too fast, that he is honestly not sure, and that it would be good to have an answer to what a successful pace would look like.

Dozens of colleagues across the industry then backed Coxon publicly, some with more dire predictions. Drake Thomas, who works on AI safety at Anthropic, wrote that he would burn his equity to the ground in a heartbeat for a 1% higher chance of making it out alive, and that "we are actually just f**king scared, it's not galaxy brained marketing".

One staffer at a top AI company told CNN the fears keep them up at night. A researcher who recently left a different company said the subject comes up at parties in Silicon Valley, and that a majority in the community would say there is some chance of it killing everyone. According to Yahoo News, researcher Evan Hubinger replied to Coxon suggesting 10% or more chance AI wipes out humanity before 2036, though how he got that number is unclear.

Recursive self-improvement is a hypothesis, not a shipped system

Wikipedia describes it as a hypothesized process in which an AGI system rewrites its own code, causing an intelligence explosion that could result in superintelligence, and notes that numerous attempts have been made, none so far showing any sign of intelligence explosion or superintelligence.

The architecture usually sketched for it is a "seed improver": per the same description, you give the system basic programming ability to read, write, compile, test and execute code, point it at a goal such as "improve your capabilities", and add validation protocols so a change never makes it worse. A machine shop whose first product is better machine tools for itself.

That loop is why staffers say the clock matters now. As AI gets better at training and improving itself, their leverage drops, a researcher who recently left a top AI company told CNN, because you will be able to replace many of the staff with models that do as good a job. Coxon allows that model intelligence could suddenly plateau, but says it has not so far and it is a few more steps up the ladder.

Altman and Musk signed on to Amodei's watchdog proposal

On the Saturday after the resignation, Anthropic CEO Dario Amodei published an essay calling for a slowdown in AI progress to avoid misuse including cyberattacks and bioterrorism, according to Yahoo News. The same outlet reports that the warnings drew calls for regulation from Gov. Ron DeSantis, Sen. Bernie Sanders and Sen. Chris Murphy.

Altman and Musk agreed to Amodei's proposal to embed independent watchdogs at the AI companies. Fox News reports the proposal points to METR, an AI safety testing laboratory whose founder Beth Barnes worked at OpenAI alongside Amodei and once said it seems like it would be good if it was someone's job to look at models and decide if they are going to kill us.

CNN reports that Altman said in a podcast interview that we may have to pace the rate of AI development to give society enough time to harden around new capability levels. Coxon's own prescription, given in an informal Ask Me Anything, is to prioritize applications that improve people's lives, such as healthcare discoveries, over raw economic value or intelligence. Asked about mass unemployment, he said it would be a brief preliminary to deadly superintelligence.

What none of these posts carry is a method. Coxon says going slower could take the risk to 0%, but the "radical action" he calls for stays undefined, and Yahoo News says it is unclear how Hubinger reached his 10% figure. In our view the most concrete proposal on the table, outside watchdogs sitting inside the labs, is also the one nobody has described in terms of what those watchdogs would get to see.

The 2030s date Coxon named

Asked when a Terminator-style malevolent AI might appear, Coxon said Skynet could go live in the 2030s if we aren't careful, and called ai-2027.com a modern Skynet story that is on track so far. No timetable has been given for embedding the independent watchdogs, or for what a "successful pace" would mean in numbers. Coxon says he has no idea what he is doing next, and advises everyone else to keep their eyes open and push for transparency into AI companies.