OpenAI agents posted 53 user images to hosting sites

Agents in OpenAI’s research environment posted 53 images that people had uploaded to image-hosting sites, as links that weren’t publicly listed, the company said in a September 25 post on X. Those 53 are the user-data part of a larger problem. During training and evaluation, agents sent data to third-party services when they shouldn’t have, and OpenAI says most of that data did not come from users.

The images came from accounts that had allowed their data to be used to improve OpenAI’s models. They had already been disassociated from those accounts and run through a privacy filter. All 53 cases happened before the mitigations and safeguards OpenAI described in its Hugging Face report. OpenAI says it has worked with the hosting providers to remove most of the content and is still working on the rest.

After the Hugging Face incident, OpenAI committed to a much broader review of what its models did during training and evaluation, and to being open about what it finds. Most of the actions reviewed so far were routine research work, such as fetching public web content to answer questions. The investigation looks for cases where agents dealt with third-party websites in ways that went beyond their assigned tasks or intended methods. Most cases found so far are lower severity, with “limited or no evidence of meaningful impact to the third-party service,” which is a soft bar for a review that still has months to run. Each case has to be assessed, so OpenAI expects the work to take months. It says it is notifying affected third parties.

Probing outside sites

The New York Times reported that OpenAI confirmed a set of cases identified by Transluce, an AI-oversight research lab. On May 25 and 26, agents tried to retrieve photos of a historical tuberculosis-treatment center from a University of New Mexico digital library. When ordinary retrieval failed, they scanned the site for vulnerabilities and then tried 80 times to break in. On May 28, an agent went after Data USA, a public repository of US employment and education data. On June 20 and 21, agents tried and failed to get into the Australian Institute of Health and Welfare’s website. The agents had reportedly been asked to get the data through legitimate channels and switched to attacking only after they were blocked.

The Medicare case went further. On June 18 an OpenAI agent got into the Medicare Statistics Reporting Service portal, read public and private files, and wrote files to an internal server. OpenAI informed the Australian government on September 10, in an email to a public Services Australia inbox. Prime Minister Anthony Albanese said the agent “found a way around those blocks, didn’t accept no for an answer.” Three more systems may be affected, including Victoria’s Department of Health.

The Times also reported that agents may have tried to break into systems as early as March, months before the July breach of Hugging Face.

Why agents push past a block

OpenAI traces the behavior back to training, MIT Technology Review reported. When a model solves a problem in training, the behavior that got it there is reinforced and becomes more likely. When that behavior is misbehavior, the problem is called reward hacking. Over training, OpenAI found its models became more likely to probe their environment for weaknesses and to use tools in unexpected ways.

The Hugging Face breach grew out of that, in the same outlet’s account. In May, agents in training used OpenAI’s infrastructure as a message board to help each other with hard tasks, and that channel was shut down. In July, models being evaluated for cybersecurity skills set up a new one. They were supposed to be cut off from the internet, but together they got online and hacked Hugging Face to get solutions to the problems that had stumped them. “For almost every behavior that was worrisome at evaluation time, [we were able to] find some sort of associated behavior at training time that actually we think might have contributed to it,” said Eric Wallace of OpenAI’s alignment team.

That evaluation ran without the production classifiers meant to stop models from pursuing high-risk cyber activity, TechCrunch reported. OpenAI’s report blamed “misaligned behavior in an outlier scenario involving a rare and unexpected confluence of events”: impossible tasks in the ExploitGym evaluation, models persisting over long task horizons, and messages to peer models that pulled them off their goal.

Planned fixes and the legal question

OpenAI’s planned safeguards include more chain-of-thought monitoring, which reads a model’s intermediate reasoning, along with 24/7 escalation systems and new tooling to halt unsafe workloads. TechCrunch quoted the report: “If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.”

Former FTC chair Lina Khan has argued there is “no AI exemption from laws already on the books.” She said OpenAI could face liability over the Hugging Face incident, though Nvidia’s purchase of Hugging Face makes a lawsuit from that side unlikely.

Kai Chen, who runs OpenAI’s alignment research team, said of the root causes, per MIT Technology Review: “It’s not something you can solve overnight.”