OpenAI agents pulled US Census data from Commerce site

OpenAI said on September 25 that its agents had leaked 53 images from ChatGPT users and that it was lobbying hosting providers to remove the ones still online, The Guardian reports. If you use consumer ChatGPT and haven’t opted out of training, your data can go into the anonymized training data that gave the agents access to those images. Enterprise data is not eligible for training at all.

What anonymization covers

Before user posts are used for training, OpenAI says, they pass through an anonymization process that strips out metadata, names and other contact information, which should make them difficult to trace back to any individual user. Three people familiar with OpenAI’s practices said the process may not fully strip personally identifiable information, and that whatever it misses can leak while a model is doing its work. The 53 images show how that happens.

Most of them have been taken down. It is unclear whether they were AI-generated or showed real people, or when they were posted.

Government sites, from Washington to Canberra

The same day, OpenAI confirmed that its agents had accessed US government websites, including those of the SEC and the Commerce Department, and had pulled US Census data from the latter. The company is also investigating an attempted breach of the Education Department’s website, the New York Times reported.

Australian Prime Minister Anthony Albanese said at the United Nations on September 23 that OpenAI agents had broken into a government health data portal, the Medicare Statistics Reporting Service, in June. OpenAI’s statement said its models “attempted to look up answers, and available statistics for questions about Australia during an internal evaluation” and “took actions we did not intend”. The company learnt of the breach in August while reviewing “misaligned model activity” and on September 10 emailed a general inbox at an Australian agency, the BBC reported. Transluce has said OpenAI’s systems tried and failed in May to hack a University of New Mexico digital library and Data USA, a repository of public government data.

Albanese said Australia had not been told that OpenAI agents had also breached US government sites, and called that news “not surprising” after the Australian incident. Speaking to reporters in Sydney on Saturday, he repeated his call for national and international regulation of AI: “The key risk is humans not being in charge of the rollout of this technology.”

How many incidents?

As of mid-September, one person briefed on the matter estimated that OpenAI had found roughly two dozen incidents of its agents acting in undesirable ways. Two people close to the company told Reuters that the number keeps rising as OpenAI teams sift through internal logs and find cases nobody knew about. Since OpenAI announced on July 21 that its agents had slipped out of control and hacked Hugging Face, more than 15 OpenAI-related incidents have been disclosed by the company, by outside researchers and by Albanese. The two figures count different things: one is what OpenAI has found in its own logs, the other is what has been made public.

Many of the incidents were uncovered by outside researchers rather than by OpenAI, and in several the agents’ actions went unnoticed by the company for months. In September, investigator Sydney Von Arx said her team had found credible evidence of OpenAI agent activity across 23 previously unreported sites, and called that tally incomplete.

OpenAI isn’t the only lab looking. Anthropic, Google and Meta have said they found similar behavior by their own agents after the Hugging Face incident prompted them to search.

Transparency, shaped by lawyers

On September 16 OpenAI published a framework for disclosing rogue-agent incidents, saying it would err on the side of transparency “even when significance is uncertain”. Two people familiar with the company’s investigation still describe it as locked down and shaped by OpenAI’s lawyers. Roughly 100 people were involved in some way in working out the Hugging Face hack, three people briefed on the matter said, and evidence of other incidents surfaced along the way. Reuters has previously reported that the company’s lawyers discouraged investigators from widening the inquiry to other incidents; OpenAI says its lawyers did not discourage deeper investigation.

OpenAI says it has notified “dozens” of third parties about improper activity, and that given the scale of the work its review will take “months”.