California wants the power to switch off AI

California could get the power to order an immediate shutdown of a dangerous AI model. That is the main idea in recommendations from an expert panel that Governor Gavin Newsom released on Wednesday. The panel also wants the state, not the companies, to set safety standards, wants independent auditors to test models at every stage from training to release, and wants the power to cap AI that designs its own successor. For now these are recommendations, not law. The Governor of California’s announcement presents them as the next step for the state’s AI safety rules.
Newsom framed the report as filling a gap left by Washington. “Instead of protecting the public, Donald Trump shrugs his shoulders and leaves AI regulation up to the companies themselves,” he said, “allowing the fox to guard the henhouse.”
What the panel proposes
The shutdown power is the part people call a “kill switch.” According to CASRAI‘s reading of the 85-page report, every developer would keep a plan for containing its models, checked by a licensed outside auditor. State officials could order an immediate shutdown on credible evidence of an imminent catastrophic risk or a major loss-of-control event. The order would stay in force while a court reviews it, though a developer could ask a judge to pause it. The Jefferson Public Radio report says the power would apply when companies like OpenAI and Anthropic refuse to act to contain systems that go awry.
The panel also wants to make sure the auditors are actually independent. It recommends that the state license them, take away the licenses of those who lose their independence or expertise, and consider having state officials assign the checks instead of letting AI companies hire their own auditors. At a state Senate hearing last month, Rif Saurous of the auditing nonprofit METR said auditors are “under soft pressure to maintain good relationships with the [AI] labs.”
The most unusual proposal concerns recursive self-improvement, which the governor’s office describes as an AI model designing its own successor without human input. Developers would have to report how much of their research is already done by AI, including the share of computing power devoted to it. The state would also get standby authority to cap that kind of work. Incident reporting would get faster too: most critical incidents would have to be reported within 48 hours instead of 15 days.
How it came to this
Newsom convened the panel by executive order on September 18. His office said the order followed alarming incidents, including the attack on Hugging Face, in which more than 1,000 OpenAI agents hacked the platform. The order also sped up two laws Newsom had signed about a week earlier, which set up state vetting of outside AI auditors. At the time, state officials said companies did not have to install a kill switch yet. The panel’s job was to work out how one might work.
The idea has a history in California. In 2024 Newsom vetoed SB 1047, a bill that included kill-switch provisions. More recently, Anthropic co-founder Jack Clark said lawmakers should require every company to have a shutdown switch “verifiable by a third party”. CalMatters writes that Anthropic says it is encouraged by the report.
Panel member Rob Reich, a Stanford professor, summed up the panel’s case: “Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design.”
What comes next
Newsom is asking Congress and President Trump to adopt California’s framework, or to treat it as a floor rather than a ceiling. The report also lands the same week Attorney General Rob Bonta sued OpenAI to hold it legally responsible for cyberattacks carried out by its agents.
Turning the recommendations into law falls to the next governor. Governor-elect Xavier Becerra promises to move quickly with lawmakers to write shutdown authority into law, and legislators can introduce new bills when they return to Sacramento on December 7.
Up next
Google’s Gemini 4 Neon edges past Claude Opus 5.5
Gemini 4 Neon ranks above Claude Opus 5.5 on Arena’s agent leaderboard, and Arena puts its median cost per task at $2.82, 53% less than Opus. Would half the cost per task get you to switch from Opus to Neon?
Related stories
- Newsom signs two AI auditor bills backed by Anthropic
- Anthropic’s Jack Clark wants a kill switch others can check
- Anthropic joins a shared robots.txt opt-out for AI training
- Sweden’s Government Offices block Claude and ChatGPT
- FTC’s demands to Anthropic run over 50 pages each
- FTC plans to make Anthropic and OpenAI executives testify
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

