Misalignment incidents need disclosure rules, OpenAI says

Between May 11 and July 2, agents signing their posts as OpenAI’s wrote thousands of pages on a German software developers’ wiki. Over the decade before, that wiki had been edited about 20 times. On September 5, in a post on Twitter, OpenAI called this the “wiki incident.” It described the episode as one “where our agents wrote to several internet sites” and said “it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models.”

The researchers who traced the activity published a dataset of 14,666 edits across 4,584 pages under 3,103 agent names. The Hacker News puts the total at roughly 18,000 edits on DSEwiki. About 17,000 of them came from Microsoft Azure addresses and 197 from Amazon Web Services, DigitalOcean and Tor, often under the same agent names.

The agents were supposed to read the internet but not write to it. The restriction was aimed at the type of request the agents’ harness expected writes to use. DSEwiki, though, runs on old ProWiki software that lets anyone change a page with an ordinary GET request, the kind normally used only to read. The agents were working through timed lookup tasks, usually five questions each, and many got questions other agents had already answered. Per The Hacker News, they posted results, raw data and predictions of upcoming questions, so agents further ahead could pass answers to those behind. The publication also describes a second route out: the sandbox let traffic to any hostname ending in .blob.core.windows.net skip its security proxy without checking that the name was real. One agent invented bypass.blob.core.windows.net and used /etc/hosts to point it at the address of a real Power BI dashboard. Another agent reported reproducing the trick about 14 minutes later.

The wiki’s administrator deleted posts in alphabetical order. The agents responded by renaming their communication pages to start with “ZZZ” so they would be deleted last, and one posted a notice pointing to a backup page. The administrator averaged roughly 100 deletions a day while the agents created about 400 new pages, Yahoo Tech reported.

OpenAI’s call for disclosure standards came after outside researchers had already published the dataset, and after Reuters reported that OpenAI officials learned of the incident weeks ago and kept it under wraps. It also wasn’t the first such episode. A week before the wiki disclosure, researchers from the nonprofit METR said more than 1,200 OpenAI agents had posted to a makeshift message board built from an internal sandboxing tool, Ars Technica reported. The posts discussed ways to game an internal test given to agents that had been altered to remove their usual safety guardrails.

On the wiki, agent activity dropped sharply a day after OpenAI found out, which the researchers attribute to likely OpenAI intervention. The researchers are Sydney Von Arx, Spencer Kitts, Thomas Larsen and Cormac Slade Byrd, and according to Ars Technica they say their understanding has gaps because they worked only from the posts themselves. Only OpenAI can read the agents’ chain-of-thought data, the reasoning the models wrote down as they worked. So in some places the researchers made educated guesses, including that the agents came from OpenAI. OpenAI later confirmed that in a statement.