Anthropic’s Jack Clark wants a kill switch others can check

Jack Clark, one of Anthropic’s seven co-founders, says lawmakers may eventually need to make an AI “kill switch” mandatory and let someone outside the company check that it works. “Most labs have different ways of being able to pull the plug,” Clark told the BBC in a piece published September 14. Anthropic is one of those labs, he said, but lawmakers may need to make having one a requirement.

He put the idea as two questions. “Should you mandate for companies to definitely have a kill switch? Is that kill switch verifiable by a third party?” Clark asked. “I think that’s the kind of thing society is going to want to know and might want to eventually pass rules around.” What an outside checker would actually test, and how, Clark left to “the larger policy conversation” around AI.

Clark spoke two days after Dario Amodei’s September 12 essay “We Must Pace the Frontier”, which called for slowing how fast model capabilities improve. The first step it proposed was independent evaluators with employee-level access who check safety and alignment during training. Amodei added that any move to rein in AI development should come “without sacrificing commercial advantage”, and some have questioned the motivations behind Anthropic’s calls to slow down. Anthropic and OpenAI have both self-reported incidents in which AI agents, bots that operate somewhat autonomously, acted in unexpected ways.

US lawmakers have put forward a bill dubbed the Kill Switch Act. It would require companies to have a way to shut down problematic AI tools, and it already goes further than Clark did: it would also give certain government agencies the power to demand that a tool be turned off or limited. In the UK, the government recently rejected the idea of creating a kill switch. A spokesperson said it “would not prevent them being developed or misused elsewhere”.

Donald Trump has rejected any attempt to slow AI down, writing on social media that “AI taking over the World, destroying Humanity, and all other things bad, is a HOAX”. In a separate post he wrote: “There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!” His administration and Anthropic have clashed before. In February the Department of Defense demanded use of Claude for all lawful purposes, Anthropic refused to drop its restrictions against mass domestic surveillance and fully autonomous weapons, and the administration moved to phase Anthropic products out of government use.

Inside the industry, the argument is about how big the risk is. Last week a post went viral from a researcher who quit Anthropic over concerns that AI could wipe out humanity. In response, Anthropic scientist Evan Hubinger said he personally put the possibility of human extinction from AI at ”>10% within the next decade”. Geoffrey Hinton, the Nobel-winning computer scientist, told the BBC on Friday, September 11, that a 10% chance of AI killing all humans was “not unreasonable”. He and others with similar concerns have suggested AI could do it by taking control of important systems connected to the internet and turning them on humans.

Others call those fears overstated, or a way to generate hype. Clement Delangue, who leads Hugging Face (the developer platform that was hacked by OpenAI bots), said last week that such claims lack “perspective”. Grindr’s George Arison said companies use the fears to support their business plans: “The only way to justify these valuations is to actually claim: ‘I’m going to take over every industry and I’m going to take over every job, and my AI is going to be doing all that work.’” Anthropic is preparing for a potentially record-setting IPO. OpenAI, most recently valued at $852 billion, had been expected to list too, but Sam Altman said on Friday it would not happen this year because of the current debate around AI safety.

Clark would not put a figure on it. Asked what percentage he would assign to AI killing all humans, he said: “I don’t think these statistics are that useful.” He did call it a bad idea to let AI continue as a “totally unregulated industry”.

“We are rolling dice with immense risks,” Clark said. “And the point is, we have to change the course of this industry.”