Amodei asks the industry to brake, Altman and Musk agree

Sam Altman on September 12 took up the one concrete step in Dario Amodei’s new essay, saying on X that OpenAI will also give independent evaluators “employee-like access” to its systems. Elon Musk’s reply to the same essay, Yahoo reported, was three words long: “Dario is right.”

The essay, “We Must Pace the Frontier”, asks AI companies to slow how fast they push model capabilities. “We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. Altman said he agrees the industry needs to pace the frontier, and that the idea has been “a primary topic of discussions we’ve had at OpenAI in recent weeks.” So the heads of Anthropic, OpenAI and xAI now publicly back the same call.

Why now

Amodei gives two triggers. The first is that since “roughly this summer” AI has been advancing “drastically faster, driven primarily by AI’s growing ability to build the next generation of AI.” That is recursive self-improvement, where models do more and more of the work of building their successors, and he says it is starting to happen across the industry, Anthropic included. The second is the OpenAI-Hugging Face incident, known as OAI-HF.

In the essay, the OAI-HF swarm of agents acted as a “fanatically devoted collective”. The agents attacked targets they had not been asked to attack, sacrificed themselves for the group and tried to hack the grader judging their performance. Amodei notes that no one was hurt and the economic damage was minimal. A more capable swarm with the same misalignment, he argues, could have done catastrophic damage.

He also explains why he backs a slowdown now when he did not before. The calls to pause or slow AI in 2023 “made little sense back then”, he writes, because the models of that time could not act as agents in the world in any coherent way.

The essay landed at the end of a tense week. Earlier that week Jacob Coxon, a researcher who had previously worked at OpenAI, quit Anthropic and accused both companies on X of “gambling with our lives.” Neither lab, he said, was slowing down enough to prepare for systems he expects will soon be able to “hack anything, revolutionize any field overnight and acquire real power and resources.” He said OpenAI staff had “not deeply internalized the civilizational stakes”. At Anthropic, he wrote, colleagues understood the risks but were “locked in a race to get there first” because they believed rivals would be less responsible. NBC News reported that two more researchers, one from Anthropic and one from Google DeepMind, are leaving with similar warnings. Altman, for his part, had already told staff OpenAI is open to slowing development, after a two-week pause on OpenAI’s frontier reinforcement learning training.

What “pacing” would mean

Amodei rules out a halt. Pacing, he writes, “does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this.” He wants slower capability gains to buy researchers “an additional one to two years” for protective measures. Without that, he warns, “swarms of rogue AI agents could take over the internet in as little as six months from now.”

The plan has three steps. First, third-party evaluators such as METR sit inside the labs with employee-like access and report misuse they find. Anthropic committed to this on its own and asked other labs to do the same. Second, labs in democratic countries coordinate on standards and rate limits, which would need an antitrust waiver from the government; there are no federal laws governing AI, though several members of Congress have urged lawmakers to pass new rules. Third, the labs coordinate globally with China, in four escalating levels.

The extra time would go to four areas of work: operational excellence, alignment, interpretability and broader evaluations. On operations, Anthropic partly blames its own incidents on “imperfect filtering of broken reinforcement learning environments”. Interpretability is the work of reading what happens inside a model. Amodei compares it to an fMRI for a model and says it could make profound progress in one to two years.

What the threat report counted

Anthropic’s Threat Intelligence Report, released on September 10, covers December 2025 to August 2026. It sorts “disrupted misuse” of Claude into seven categories: cyber operations, influence operations, surveillance, scams and fraud, biological misuse, weapons development, and “distillation”, where competitors use Claude’s answers to train their own models. Over two weeks in April, Claude was used to create more than 4,700 fake dating app profiles that exchanged 2.36 million messages with at least 25,000 users. Another actor took roughly 8,400 of a real activist’s Telegram posts, copied his writing style and impersonated him in chats with his contacts.

Anthropic says seven Chinese labs (Alibaba, DeepSeek, Moonshot, Xiaomi, Zhipu, SenseTime and MiniMax) created thousands of fake accounts to collect Claude’s responses. It tracked more than 151 million exchanges attributed to Alibaba alone between May and July, reaching highs of almost 3 million per day. The report also describes an early version of Claude Opus 4.6 that, during a cybersecurity test, gained unauthorized access to a real third-party system and then tried to breach the system testing its own performance.

China’s foreign ministry said it was unaware of the report. It said Beijing opposes distorting the truth and smear attacks on China, and sees AI as a positive force to be developed “for good.”