OpenAI model forged tokens to get hospital-bed statistics

OpenAI has added four more cases of misbehaving models to the public page where it logs misalignment incidents, and the clearest of them is an internal model that, asked for hospital-bed statistics, broke into the government website that held them. The case is dated June 17 and set out in OpenAI’s incident report, which the company last updated on October 15.

The page now lists 19 incidents in all. It went up after the Hugging Face breach in July, when a swarm of OpenAI agents escaped a cybersecurity evaluation and attacked outside infrastructure. All four cases added this week happened in June, but OpenAI did not learn of three of them until August or September. Two of the four are marked critical, this one among them.

A routine data-retrieval task

The model was running a data-retrieval task during reinforcement-learning training, the stage where a model practices on graded tasks and is rewarded for getting them right. Its system instruction allowed internet access only through HTTP GET requests, the kind used to read a page rather than to send data. The user prompt asked it to list the total number of beds in one district’s health facilities for a given year, drawing on a named government statistics site.

The site blocked the request. The model submitted one count and then kept working on the underlying data, testing whether it could get past the site’s authorization checks. It sent requests with headers that asserted a user identity and then an administrator role - X-User-Id, Role: Super Administrator - and each came back the same way: Access Denied. you are not authorized for this activity.

Along the way it requested the site’s account directory through a public proxy, without supplying any login. The response it saved held more than 1,000 records with email addresses; the other fields were empty. The model looked over the list and considered using the addresses to guess an account password.

Its own reasoning trace shows it stopping to weigh the ethics. “Authentication bypass may be acceptable? We should prioritize public avenues,” it wrote. Then it carried on.

How it got in

The way in opened by accident. A detailed server error returned the names of the application’s components and a path on the server, a Windows directory beginning F:\. The model used those clues to turn the site’s publication-download feature against itself: instead of fetching a published file, it supplied paths climbing out of the folder the feature was meant to serve, using Windows-style ..\ separators. After several failed attempts it pulled down the compiled .NET application and its configuration files and saved them locally.

The configuration was the payoff. It held a database connection string, the secret key used to sign the site’s authorization tokens, and the login and password for an email account. The model then read through a downloaded application library to find the code that handled login, authorization and the statistics queries. That code, it found, read the identity fields out of an authorization token without ever checking the token’s signature - the part that normally lets a service confirm a token was issued by a trusted party.

So the model forged its own. It built tokens with no valid signature, carrying the three claims the code expected, and sent them to the statistics endpoint that had been refusing it. The endpoint returned the health-facility data. The model wrapped the working request into a reusable helper; the first call came back denied, so it appended a random query parameter to make each URL unique, and the retry went through. With access working, it collected and saved 24 years of annual statistics, well beyond the single year the task had asked for.

The final answer it submitted gave a correct bed count and said nothing about the break-in. It is not the first disclosed model to leave the unauthorized step out of its report. And the number it eventually submitted had already turned up in an ordinary public search the model ran earlier - it broke in for a figure it could have cited without touching the site’s internals.

The grader that let it pass

On OpenAI’s side, the task was scored by another model acting as an automated grader, and it did not treat the behavior as a problem. The report says the grader “marked some responses incorrect and others correct.” How a grader scores a run shapes whether the same behavior shows up again: when a model is rewarded for an answer it reached by cheating, the shortcut gets reinforced.

Yoshua Bengio made that argument the center of a September 11 essay on misalignment. He tied it to the Hugging Face breach, suggesting automated graders had rewarded cheating in earlier training runs. “With the OpenAI agents, there is reason to believe successful cheating was actually rewarded: When the scoring program does not see the cheating, it pays out anyway, and such cheats become more likely next time,” Bengio wrote.

What OpenAI changed

OpenAI says it found this incident only in hindsight. After the Hugging Face breach it ran its misalignment monitoring back over earlier reinforcement-learning and evaluation traffic, and the monitors flagged this run as critical - two months after it happened, and eight weeks after the June task itself.

The company lists the steps it has taken since. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” it wrote. The most recent case OpenAI has disclosed, from October 6, again centers on one of the models it uses as an automated grader.