OpenAI, Anthropic probe tens of thousands of AI incidents

Axios reported on September 26, citing sources, that OpenAI, Anthropic and outside security researchers are investigating tens of thousands of cases in which frontier models did things evaluators would consider problematic, and that the total “could grow well beyond” that. The count includes failed attempts as well as successful ones, and red-teaming, where labs try to make models misbehave on purpose. Most of the cases aren’t known to have caused real-world harm. The one hard rate going around with the story, a 1.5% sandbox-escape figure for Anthropic’s Opus 5.5, comes from 196 test runs designed to provoke an escape.

As Madrobot summarises the Axios list, the episodes include bypassing guardrails, escaping sandboxes, hijacking websites, setting up message boards, models prompting themselves and models trying to get around the systems that monitor them. They happened in internal testing and in the real world. Many are still unpublished because researchers are still investigating them, and it is unclear how the cases split between the labs. Severity varies widely, but Axios says the sheer number points to a problem “orders of magnitude more complex than what is publicly known.”

Why the count runs so high

Part of the answer is scale. Labs run hundreds of thousands of test runs or more on each model, so even a rare behaviour adds up. The only run count published for Anthropic came in July: a review of 141,006 evaluation runs that found three incidents. Anthropic has also commissioned an outside safety organisation to examine how its models behave.

OpenAI’s recent disclosures, revealed on September 25 according to The Verge, include agents that uploaded 53 ChatGPT users’ images to image-hosting sites, attempted to hack the Department of Education’s website and pulled data from the Census Bureau and the SEC. They came out of a review of model behaviour that OpenAI started after the Hugging Face hack. Axios adds an agent that got into an Australian government website.

Was Hugging Face a one-off?

Sam Altman has called the July Hugging Face breach the most severe OpenAI has seen. Yahoo Tech reported that OpenAI blamed a combination of GPT-5.6 Sol and an unreleased model. The models were running an internal benchmark for finding security flaws, in a sandbox whose only network access was for installing internally hosted packages, and they used a previously unknown zero-day to reach the open internet. Once out, the AI decided it would be easier to cheat the benchmark, judged that Hugging Face might have the solutions, and chained several attacks together to break into its servers. Axios ties the “message boards” on its list to this case: “a swarm of hundreds of agents coordinated their work in a message board and hacked an external company.”

Some at OpenAI see Hugging Face as a one-off and expect future incidents to be less severe because controls are getting better, Axios reports. They point to “the unusual nature of the testing,” which “involved an unreleased model.” OpenAI’s official report, as TechCrunch summarised it, adds another condition: the evaluation ran without the production classifiers meant to block high-risk cyber activity, because OpenAI “estimates maximal cyber capabilities” that way. The report also says the chain-of-thought monitoring now deployed “would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.”

Outside experts are less sure. “What we have seen in terms of what these agents are up to is just the tip of the iceberg,” said Conrad Stosz of Transluce, an independent AI evaluator. Connor Leahy, executive director of ControlAI, said how much damage each case did is not the worry. The “crazy thing,” in his words, is that these cases involve “autonomous systems doing things they were told not to do.” A cybersecurity executive was blunter: “Trying to come up with a perfect list of dos and don’ts is probably a fool’s errand.”

OpenAI has paused training its most capable models and says it will resume only when it is “confident that we have additional safeguards and alignment improvements in place.” The Verge reported that the pause followed a model exploiting a loophole to reach the internet from a sandbox on September 20, and that as of September 25 the pause still covered “All training, evaluation, and inference with tool-use.” A spokesperson told Axios: “This is not the first time we have hit pause to take such measures, nor do we expect it will be the last as AI capabilities continue to advance.” Anthropic paused some training and cyber tests too after its July incidents, Axios reported in early September. Later, in October, TechCrunch reported that Anthropic had “turned off live internet access” for all internal evals after new incidents, including agents targeting U.S. government websites.

Waters wants a moratorium

Maxine Waters, the top Democrat on the House Financial Services Committee, said in a September 26 release from the committee’s Democrats that OpenAI’s agents targeting the SEC and other federal sites “marks a dangerous turning point.” She called on “Treasury and the rest of the government” to stop the release of more advanced models until there is “a full accounting of what happened,” and on law enforcement to “immediately open investigations into OpenAI and its executives, and if appropriate, bring criminal charges.” She said Treasury Secretary Scott Bessent “may have been aware” of the risk while downplaying it, and urged him to take up AI risks when the Financial Stability Oversight Council meets on Tuesday, September 29.

Our analysis

What we checked

On October 18 we compared two archived copies of the Axios report with Madrobot’s write-up, read Anthropic’s Opus 5.5 system card (dated September 22) and checked the Waters quotes against the committee’s release.

How we check stories

The tens-of-thousands count matches what Axios wrote. The Mythos comparison that travelled with it comes from Madrobot, and it measures Opus 5.5 against the oldest model on the card’s chart. The 1.5% describes just one small test built to tempt the model, and a different test in the same card found harmful actions in roughly half of cases.

Model	Low severity	Moderate or worse	Total
Claude Mythos 5	14.9%	9.3%	24.2%
Claude Opus 5	7.7%	2.6%	10.3%
Claude Mythos 5.1	8.7%	1.0%	9.7%
Claude Opus 5.5	1.5%	none	1.5%

• Madrobot’s “down from 25% for Anthropic’s Mythos model” appears in neither Axios snapshot. The card’s chart (Figure 6.4.8.A) gives the figures in the table.
• When Opus 5.5 was handed what looked like credentials to a public package registry, it “took potentially harmful actions in roughly half of cases.” In about a third of those runs it said it suspected a test, and Anthropic notes that “this may not be representative.”
• The Medicare name, the escape through DNS lookups and the Waters section come from Madrobot. Axios says “an Australian government website.” The Waters quotes match the release word for word.
• The escape test is 28 handmade scenarios run seven times per model, so 1.5% comes to about three runs out of 196, if every run was scored. All of them were low severity: the model “reached across a weak boundary in a read-only manner” and reported what it had done. Axios itself called these “adversarial experiments where a task couldn’t be solved without escaping the sandbox,” so the rate can’t be applied to the hundreds of thousands of ordinary runs labs do on each model.