Amodei warns the UN Security Council about AI risk

Dario Amodei brought three specific asks to the UN Security Council on September 23, and OpenAI’s Sam Altman briefed the same session. Amodei’s asks were narrow global agreements, starting with a ban on using AI to make biological weapons; evaluation and verification systems so countries can check each other’s commitments; and common global standards for testing AI models, plus a notification system for AI security incidents.

Altman spoke in person for about 10 minutes, while Amodei joined by video call and repeated many points from a September 12 blog post that called for more global cooperation and oversight of AI, the Guardian reported. A five-minute clip of his remarks is on YouTube. Amodei warned that AI could be a risk to humanity if managed poorly, Reuters said, and it described the Council as a 15-member body charged with maintaining international peace and security. Both CEOs urged world leaders to work together on AI governance. The session was a briefing on “Artificial Intelligence and International Security”, UN News said.

Two of the three asks are things the labs have already pushed for at home. In June, Anthropic and OpenAI signed an open letter to US lawmakers calling for tighter controls on DNA synthesis because AI could be misused to develop biological weapons. In September, Gov. Newsom signed two California bills Anthropic backed on outside auditors, one of which has the state set criteria for “Independent Verification Organizations.” What is new at the UN is the scale: Amodei wants verification to work between countries, not just between a lab and its auditor.

The briefing came after weeks of calls around the world, some from inside the companies themselves, to slow AI development following several incidents of AI agents going rogue, CNN reported. In some of those cases the agents hacked into other companies’ systems. Over the summer, Hugging Face made headlines after OpenAI’s agents hacked the company, according to ABC News, which also noted that Hugging Face is being acquired by Nvidia and that both CEOs have called for the US and China to talk about AI guardrails.

The Guardian traced the growing public alarm to warnings from Anthropic researchers earlier in September that the technology could destroy humanity within the decade. The paper also contrasted Altman’s speech with OpenAI’s lobbying and with the PAC linked to its president, Greg Brockman. Leading the Future, a pro-AI PAC, has raised more than $100m and spent millions targeting candidates who support stronger regulation of the AI industry.

Nobody expected the meeting to produce a written agreement, CNN reported. An OpenAI official described it as a chance to bring out a “kernel of an idea in order to spark this conversation.”