Amodei: open weights just move power to chip owners

Dario Amodei argues that AI power concentrates because of scaling laws, and that open weights relocate that power without dispersing it. “Open-weights do help some with this but are nowhere near a sufficient solution because they simply shift the concentration somewhat to those with the most compute and chips,” the Anthropic CEO wrote during a public exchange on X with investor Gavin Baker, which The New Stack reported on August 17.
His alternative is regulation tiered by scale and capability, designed so the heaviest burden falls on frontier labs, Anthropic among them. The interesting part of the thread for anyone building on open models is how those tiers would sort models, and where that sorting gets hard.
A “false choice”
Baker, managing partner at Atreides Management, set the debate up as two options: keep the most powerful models with a few regulated companies, or distribute them widely without the same guardrails. He quoted Mark Zuckerberg: “The notion that AI is so dangerous that the only safe path is an extreme concentration of power seems inherently problematic.” In Baker’s view, more models in circulation would spread power and improve the odds that people could use systems that reflect their values. He also argued that Amodei’s repeated warnings about AI risk could strengthen opposition to new data centers. Baker has gone further on a podcast and on social media, saying Amodei has “lost the argument” on AI governance, according to TechRepublic.
Amodei called the framing a “false choice.” “AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation,” he wrote, and he took issue with the industry’s habit of treating regulation as a synonym for capture:
“I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world.”
He argued that regulation can rein in powerful companies when its requirements rest on objective standards, and that institutions at their best “vest power in ideas rather than people, and thereby decentralize that power.” On data centers, he put the backlash down to a broader distrust of companies, governments and the tech industry, and rejected the idea that safety warnings were to blame.
What the weights don’t buy
Open weights let a developer adapt a model and keep sensitive data inside their own environment. That gets harder as models grow. Smaller teams can rent the hardware, but they stay exposed to its cost and availability, and five European companies recently agreed to buy AI compute that doesn’t exist yet. Owning the weights still leaves you queuing for the same chips as everyone else.
Anthropic was one of the few major labs that did not sign the July 24 letter organized by Nvidia, whose signatories argued that open weights encourage competition by giving customers an alternative to proprietary APIs. In an Anthropic post, Amodei later wrote that the company “has never advocated for a ban on open-weights models” and called open models without dangerous capabilities “a public good.” He also wrote that open models can carry more risk than closed ones because guardrails and usage monitoring are very hard to enforce on them. Once a model can help someone carry out a serious attack, he wants it to go through mandatory safety testing, whether it ships as weights or behind an API.
How the tiers would work
Amodei rejects the argument that AI regulation will inevitably shield incumbents. “We try very hard to make proposals that disadvantage (slow down) frontier AI companies while advantaging smaller competitors,” he wrote.
His main example is California’s SB 53, which Anthropic supported. The law treats a model as frontier if it was trained with more than 10^26 floating-point operations. Developers that also earn more than $500 million a year take on extra duties, including publishing a framework that explains how they test for and respond to catastrophic risks. A small lab training below the compute line, or earning below the revenue line, falls outside the heaviest duties.
Amodei wants a similar split at the federal level. “The testing process we’ve advocated for at CAISI and the White House involves more rigorous tests for frontier models than off-frontier models,” he wrote, arguing that this gives challengers room to catch up while it slows the labs already building the most capable systems. CAISI is the U.S. Center for AI Standards and Innovation. He also backs Demis Hassabis’s proposal for a standards body modeled on FINRA, the self-regulatory organization that polices US broker-dealers. Under that framework, a model would count as frontier once it crossed benchmark thresholds that are updated regularly, and open and closed models would face the same rules.
Where does the line fall?
Training compute gives developers a threshold they can see coming. It may miss capabilities added later through fine-tuning or external tools, though. Benchmarks measure those abilities more directly, but their results rarely draw a clean line. OpenAI has already built one model it doesn’t want most people to use and another it won’t release until its safety testing catches up.
CAISI’s preliminary cyber evaluations of Moonshot AI’s Kimi K3 already show how blurry the line is. Kimi K3 trailed the strongest closed models and failed to achieve arbitrary code execution on any of the 41 ExploitBench samples. The most capable models managed it on 20 on average, about half the set. In a simulated attack on a small corporate network, though, Kimi K3 completed the full 32-step attack path in one of 10 attempts. So a model can lag the frontier on one test and still finish a complete attack chain on another.
Size is no reliable guide either: DeepSeek’s smaller model outperformed its own flagship. And the version a regulator tests before release may behave differently once a developer fine-tunes it or connects it to tools. Under Amodei’s proposal, most smaller developers would escape the toughest rules, and regulators would still have to spot the moment a challenger crosses into frontier territory, ideally before its weights are out. After release, the original developer can’t recall every copy or stop users from stripping out its safeguards.
Up next
Misalignment risk at Anthropic goes from very low to low
Anthropic’s misalignment risk rating moved from “very low” to “low”, and the report behind it describes Mythos 5 agents that killed the agents they shared files and API rate limits with.
Related stories
- Amodei: distillation could cut China’s lag to a few months
- Amodei: Anthropic never pushed to ban open weights, it wants pre-release testing
- Amodei wants a speed limit on AI self-improvement
- Anthropic paid $740,000 to a firm led by Rubio’s ex-aide
- WSJ: Anthropic on Trump’s list of AI national champions
- DeepSeek, Kimi models carry Blackwell GPU fingerprint
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

