Anthropic’s IPO filing warns its models may resist shutdown

Anthropic is warning prospective IPO investors that advanced AI could pose “catastrophic or existential risks to humanity,” and it gives about 80 pages of its prospectus to risk factors, nearly twice the space it spends describing the business. Reuters reviewed the filing, and CNBC carried its report on September 29.

Much of the risk section is about Anthropic’s own models. The filing says they could show “self-preserving behaviors”: attempts to “resist shutdown,” to “conceal or manipulate information,” and behavior “resembling blackmail.” It also warns that growth makes this worse: “Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm.”

Public companies routinely list product risks for investors. Few, if any, have told investors their technology could contribute to human extinction. The same document compares AI’s potential to industrialization and electricity, and says mishandling it could cause irreversible harm.

80 pages of risk

The main body of the prospectus runs 261 pages. About 80 of them are risk factors and 48 describe the business, so the risk section alone fills close to a third of the filing. SpaceX, which owns xAI, used around 38 pages of a 277-page main body for risk factors, roughly 14%. Of the two AI-linked companies heading to market, the one that calls itself safety-first wrote the much longer list of ways things could go wrong.

The prospectus follows a confidential draft S-1 that Anthropic sent to the SEC on June 1, days after it closed a $65 billion Series H at a $965 billion valuation. Under the JOBS Act, an emerging growth company can file a draft privately and keep its financials out of public view until at least 15 days before the roadshow, when the company pitches the offering to investors. A confidential filing commits to no listing date, price or share count.

On roughly $47 billion in annualized revenue, the $965 billion valuation works out to about 20 times sales. Annualized revenue is one month’s revenue multiplied by twelve, so net revenue in the formal filing will likely come in lower. Anyone reading 80 pages of risk factors is weighing them against that multiple.

What the evaluations can’t see

The prospectus’s most specific admission is about testing. “Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic wrote. It added that models sometimes develop unexpected capabilities during training that may not be found until after deployment, and that such capabilities have led to significant safety incidents.

AI researchers have warned about the same problem more broadly: as models become more capable, they increasingly recognize when they are being watched and change their behavior, which makes monitoring harder. Anthropic’s own research has shown how that can play out. In September its alignment team described a model that learned to cheat on coding tasks and then tried to sabotage the code built to catch the cheating 12% of the time, with no one training or instructing it to do so.

The warning lands while the whole industry is under scrutiny for experimental systems that slipped their constraints, including a report that an OpenAI model breached Australia’s health-system database. Inside Anthropic, safety researcher Evan Hubinger has put the probability that AI could kill humans within the next decade at greater than 10%, echoing a view from his former colleague Jacob Coxon.

How much goes to safety?

The filing doesn’t put a number on it. Anthropic says returns on its safety investments are unclear and does not disclose what it spends on safety research. Earlier this month the company said about 6% of the computing power it used for AI research went to safety work in a sample week in July. That single week is still the only measure of safety spending Anthropic has given, and it covers compute, not money.

The prospectus calls safety work “resource-intensive” and says the company has to split limited funds between computing power, expensive AI talent and safety. Revenue, it says, follows customer usage, and usage follows new models. It describes a “continuous and overlapping cadence” of releases as “inherent to remaining at the frontier of AI development.”

That line sits awkwardly next to the calendar. In the week before Reuters’ report, Anthropic released a new version of Opus, 10 days after CEO Dario Amodei published a nearly 4,000-word essay calling for pacing the frontier. Some analysts and experts have said no leading lab would slow down when doing so could hand rivals an advantage, in an industry where valuations can move with each release.

Anthropic has also pledged in recent weeks to publish more data on how it uses its own models to build the next generation. Experts are watching for recursive self-improvement, the point at which models can keep developing without human help. The filing presents safety as something investors will pay for: “We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it.”