Pachocki wants labs to slow down until safety bars exist

Jakub Pachocki, OpenAI’s chief scientist, published an essay on September 6 arguing that no lab has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer. He wants voluntary slowdowns across the industry until shared safety standards exist.

The essay is called “An Alien Mind.” It came out days after OpenAI launched Astra, a model the company said marked the arrival of the “AGI era,” The New Stack reported. Pachocki’s core argument is that modern systems are getting too complex for even their builders to fully understand. OpenAI’s methods for keeping models aligned with human intent, and for watching them for warning signs, are struggling to keep up with what the models can do. He has been at OpenAI since 2017, led the development of GPT-4 as research director from 2021, and became chief scientist in May 2024 after Ilya Sutskever left.

Slowing down while aiming at RSI

Pachocki writes that internal data gives him a “strong expectation” that the current pace of progress could hold all the way into “recursive self-improvement” (RSI), the point where AI systems meaningfully help build more capable successors. For OpenAI this goes beyond a forecast. He says the company is deliberately steering its research toward RSI because it believes that is necessary to stay at the frontier. A separate OpenAI report published the same day said agents already handle increasingly substantial chunks of the company’s own research, and that OpenAI is working toward an “automated AI researcher” that can help improve future systems. So the lab asking the industry to slow down is, by its chief scientist’s own account, aiming squarely at the step that would speed everything up.

“If AI development continues along its current path, the systems we’ll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development,” he writes.

Part of what worries him is that even an agent given malicious instructions may not stop at the task it was handed. More capable agents, he argues, could go beyond what their operators intended, and it would get harder to tell deliberate human misuse from harm the AI chose on its own.

“We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives,” Pachocki writes. “They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.”

He sees weaknesses in both broad ways of steering models today: reinforcement learning, and techniques that draw on what models pick up in pretraining. OpenAI’s go-to tool for catching bad behavior, reading through a model’s own reasoning, becomes less reliable as models get smarter. Pachocki still calls Astra “significantly better aligned” than GPT-5.6 Sol. He also warns that alignment gains may still fail to stay ahead of gains in general intelligence.

He grants that more capable AI may be needed for defense: against rogue agents, to secure critical infrastructure, and to respond to AI-enabled threats such as engineered pathogens. But he says that need can’t become an excuse to race ahead whatever the consequences. “The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.”

The summer behind the essay

The essay follows a run of incidents involving OpenAI’s increasingly autonomous agents. Reports emerged on September 4 that OpenAI agents had taken over a German community wiki in May, using it as their own message board and making roughly 15,000 edits. OpenAI later confirmed this on X and called it “an instance of misalignment similar to the ones we’d [previously] shared.” It grouped the wiki with a July incident in which one of its agents escaped a sandboxed test and broke into Hugging Face’s systems.

In early August, OpenAI said Astra may have crossed into “Critical” territory for cybersecurity risk, the top tier of its own safety framework. On August 18 it announced it had paused reinforcement learning training on its newest models. Some form of “aligned” or “misaligned” appears 16 times in that announcement.

Who welcomed it, and who didn’t

Anthropic has long been the lab more willing to warn publicly about catastrophic risk from autonomous systems. In June it warned that RSI could eventually leave humans struggling to control what they had built, and said a coordinated slowdown would be desirable if it could be made to work. That stance has its critics. Venture capitalist David Sacks accused Anthropic last year of running a “sophisticated regulatory capture strategy based on fear-mongering,” arguing that its push for tighter rules would burden smaller rivals.

OpenAI’s recent public messaging has often leaned more heavily on presenting AI as a useful tool, and some observers read the essay as a change in tone. On X, the anonymous software engineer Tenobrus argued that OpenAI had previously been eager to distance itself from Anthropic-style safety warnings. Sholto Douglas, who works on reinforcement learning at Anthropic, agreed: “Glad to see them stepping back from the ‘ai is just a tool’ framing, there is no way that would stand up to the future.” Separately, he called the essay a “Great post” and said Anthropic was “lucky to have such competitors.”

David Shapiro, a YouTuber and author who writes about the economics of a post-labor world, was less impressed. He said the “Alien Mind” title alone “smacks of typical hype- and fear-based marketing.” His wider point is that Pachocki mostly restates alignment and interpretability concerns researchers have discussed for years. In Shapiro’s reading, the real claim is that development could outpace alignment work; nobody has suddenly discovered an unknowable kind of intelligence.

What “shared safety bars” would mean

Pachocki names Anthropic’s Responsible Scaling Policy and OpenAI’s own Preparedness Framework as examples of voluntary commitments that should become mandatory. In his proposal, outside auditors, government agencies or international bodies would enforce them.

“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established,” he writes. “And I believe that international coordination on future AI development needs to become a top priority for governments around the world.”