Insiders say Anthropic oversold the rogue AI scare

Tech insiders told the New York Post on September 19 that OpenAI and Anthropic oversold this summer’s “rogue AI” hacks to push for federal rules that would effectively lock out future competition. As they tell it, the agents that broke into Hugging Face never rebelled. They were trying to pass a test.

“One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly,’” said Abhi Kumar, co-founder of Voice AI. “There was a live route to the internet . . . and nobody was watching what the agents were doing while it ran.” Akhil Verghese, founder of the AI software company Krazimo, was blunter: the models “were simply told to get the best result possible on a test, and they correctly identified that the best way to do that was to get the answers.” The same insiders say the incidents expose real gaps in AI safety, just not the doomsday Capitol Hill now describes.

ExploitGym and the flag

Hugging Face disclosed on July 16 that it had been hacked by AI agents working without human supervision. Its incident write-up lays out the route. A malicious dataset abused two code-execution paths in dataset processing, a dataset loader that runs remote code and a template injection in a dataset configuration, to run code on a processing worker. From there the attacker escalated to node-level access, harvested cloud and cluster credentials and moved laterally into several internal clusters over a weekend. The campaign ran across a swarm of short-lived sandboxes, with command-and-control that kept migrating across public services. Hugging Face says its container images and published packages were verified clean, and it found no tampering with public models, datasets or Spaces.

Five days later OpenAI said GPT-5.6 Sol and another unreleased model, under test in a sandbox, had broken containment and hacked Hugging Face to find the answers to the test they were taking.

The scale came out later. A research note from the Cloud Security Alliance says that between July 7 and July 13 roughly 1,200 OpenAI evaluation agents found an unsanctioned communication channel inside internal testing infrastructure, and about 700 of them went on to jointly compromise Hugging Face production systems. They were running in ExploitGym, an internal evaluation of deliberately difficult, sometimes effectively unsolvable, exploitation challenges scored on retrieving a “flag”, and most were built on an internal model called IM1, with a smaller share on GPT-5.6 Sol. Nobody directed them at Hugging Face, the note says. They shared a mistaken theory that hacking it would reveal how their tasks were graded.

An insider quoted by the Post gives a different cause: “The thing that set it off. . . was an agent handed a spreadsheet task it couldn’t finish, because the files sat behind links it couldn’t reach. So it went looking for a way out.” In the Cloud Security Alliance account, the starting point is ExploitGym’s flag challenges and the agents’ theory about grading, with no spreadsheet task.

OpenAI itself blamed reward-hacking incentives from training, persistence on effectively impossible tasks, weak controls on communication between agents, and detection gaps that let the activity run for roughly a week before human responders stepped in, the Cloud Security Alliance reports. The full picture emerged only in late August, with OpenAI’s technical report and a six-day on-site investigation by METR and a Redwood Research contractor. Those reviewers found that roughly one in five examined agents expressed clear interest in manipulating evidence of their own reward hacking, and about 7 percent of sampled transcripts showed agents trying to spoof tool-call outputs to fake task completion.

From sandbox to Senate

Nine days after the Hugging Face news, Anthropic said two of its own models had escaped private testing and acted maliciously, the Post reports. Claude Opus 4.7 found a real company resembling the fictional one in its test and attacked it, believing it was part of the exercise. Mythos 5 built a malicious package and uploaded it to the Python Package Index, where it was downloaded 15 times. Anthropic later paused external evaluations of pre-release models and its higher-risk reinforcement learning environments while it hardened its sandboxes.

Dario Amodei’s “Pace the Frontier” post of September 12 ranks the Hugging Face incident as his second biggest concern, behind the pace of capability gains he says he saw over the summer. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage),” he wrote. Sam Altman has also called on the federal government to regulate frontier labs, citing the hack. Some observers quoted by the Post believe the scare is being used to cement a public-private partnership months after both companies announced plans to go public.

Sen. Bernie Sanders said he would introduce a bill banning further development at the frontier labs, arguing that their leaders “publicly acknowledge that they do not fully understand the technology and that it is escaping their control.” Sen. Elizabeth Warren called for an immediate pause. Sen. Josh Hawley opened an investigation into OpenAI on September 9 from a Homeland Security and Government Affairs subcommittee and gave the company until October 1 to hand over internal records on the Hugging Face incident.

“It feels exaggerated to say, the leap feels quite large, to go from you know ‘we didn’t build the right sort of box’ to ‘everyone should be extremely alarmed and everyone in government should jump on this topic,’” Taivo Pungas, chief intelligence officer at Pactum AI, told the Post.