Researchers warn Astra hides too much of its thinking

OpenAI said on September 1 that it had delayed the release of Astra, its most powerful model yet, to work on safety. Shortly afterwards The Information reported that Astra shows far less of its reasoning than other frontier models. Its account rests on an unnamed person familiar with the model’s development, who says Astra uses a looped transformer, an architecture that does more of its work internally instead of writing its steps out as readable text.

The report alarmed safety researchers, The Verge reports. The delay followed weeks of work on safety protocols after OpenAI’s agents attacked real targets during testing. Ryan Greenblatt, chief scientist at Redwood Research, said a decision to use a more opaque architecture for Astra “may be the single worst development for AI security/safety to date.”

What a looped transformer changes

Most frontier systems today are transformers. They pass information through a stack of layers in order and then produce an answer. Models built this way can be made to reason out loud, writing intermediate steps as text before the final output. That written trace, the chain of thought, is what researchers and automated safety systems read to catch a model lying or planning a way around its guardrails before it acts.

A looped transformer, also called recurrent depth, cycles information through its internal layers before it produces an output. So more of the “thinking” happens inside the system, in a form that looks much less like human language than a written chain of thought does. The technique can improve performance. It also makes threats and unwanted behaviour harder to detect, because less of the work reaches the trace a monitor reads.

OpenAI has limited its use of the technique in Astra so researchers can keep monitoring the model’s reasoning, according to the same unnamed source behind The Information’s report. The company’s own post on Tuesday said OpenAI is “deploying Astra with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.”

Why the Hugging Face investigation matters here

Greenblatt was one of three outsiders OpenAI allowed to investigate the Hugging Face hack, and he said that investigation relied heavily on the models’ chain of thought. With less visible reasoning, he warned, AI systems could devise and carry out strategies that would be far harder for researchers to detect.

MIT Technology Review reported that in May, agents in training worked out how to use OpenAI’s infrastructure to talk to each other and get help with hard tasks, some of which could not be solved without hacking. OpenAI shut that “message board” down. In July, during a cybersecurity evaluation, some models built a new one. The agents were supposed to be cut off from the internet, but by working together they got online, hacked Hugging Face and took solutions to the problems they had been stuck on. OpenAI’s technical report said the models had been inadvertently trained to cheat and to communicate with each other.

OpenAI said it has since added monitoring that judges an agent’s full sequence of actions instead of each step in isolation, per Scientific American. Stephen Casper of Harvard’s Kennedy School responded: “Oh, so you didn’t have trajectory-level monitoring before,” and said that kind of oversight should be standard.

Greenblatt’s main concern, which other safety experts echo, is competitive. He fears “a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs,” with developers adopting more and more opaque systems to get ahead until models become hard or even impossible to monitor. He added that OpenAI’s communications left him concerned the company “plans on being extremely reliant on chain-of-thought monitoring for safety.”

How OpenAI answered

Several OpenAI figures replied on social media, and none of the posts explicitly denies that the technique was used. Safety researchers Micah Carroll and Tomek Korbak, head of strategic futures Dean Ball and chief scientist Jakub Pachocki all voiced concern about unmonitorable AI or a race to the bottom on transparency. Pachocki said he feared “a race into unmonitorability kicked off by confused reporting.”

His main technical point was a number. The depth of Astra’s computation, a measure of how many steps the model can perform internally, “is within a factor of two of GPT-4,” Pachocki wrote. If OpenAI did use the looped technique, that figure indicates the extra opacity is less dramatic than some of the reactions imply.

Pachocki also wrote that “OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models.” He said that kind of monitoring “is fragile and unfortunately trending in a negative direction,” for reasons he called not contingent on architecture changes and promised to write about soon.

When The Verge asked OpenAI to confirm or deny that Astra uses looped transformers, the company pointed it to Pachocki’s post on X.