Huang to Amodei's brake analogy: run as fast as you can

Two CEOs took the same Dreamforce stage in San Francisco on Tuesday and gave opposite answers to the same question: Dario Amodei said a rival carmaker's brake failure should make every carmaker stop and check its own record, and Jensen Huang, not long after, said run as fast as you can, according to the Guardian. Sam Altman spoke later the same day and objected to how the whole argument has been framed.

At a glance

• Amodei is asking for three things: outside evaluators embedded inside AI companies, safety standards coordinated among Democratic countries, and eventually broader global coordination on security.
• Huang argues new regulation is unnecessary and that firms should simply hold a product back until they know it is safe; he also called AI job destruction completely nonsense.
• Anthropic has committed to independent evaluators, but on the two coordination pillars Amodei offered only a dialogue with the rest of the industry, with no timeline named.

If you have not been following: Jacob Coxon, a former Anthropic researcher, resigned with a public declaration that AI could lead to the extinction of humanity within the decade, and the warning moved outward into Silicon Valley, the regulators and the public. According to the Guardian, Amodei published his pacing proposal, titled "We Must Pace the Frontier", at the weekend, and Altman, Google DeepMind's Demis Hassabis and Elon Musk each endorsed it publicly.

Amodei told the room to look at its own record first

Amodei's analogy was automotive. When a competing car company has a safety incident like brake failures, he said, it is a moment for all car companies to stop and review their own practices, not a moment to score points off the company that failed.

It's very tempting to attack your competitor and say these guys are unsafe. But I think the more responsible way to respond to it is to say, let's look at our own record. We may not have had this big, high-profile incident, but I'm sure we're not perfect.

What has surprised him most, he said, is the pace, and not only the technology's. Anthropic had a prediction of how capable the models would be, but did not appreciate that this would make the companies themselves grow so fast.

Huang's answer came down to four words: run as fast as you can

Huang appeared at the same convention on Tuesday and rejected the premise. New regulations are not necessary, he said; companies should wait to release products until they know they are safe rather than begging the US government to intervene. He also sketched a future in which every company and every enterprise becomes an AI company, and repeated that firms which do not embrace the technology get left behind.

On jobs he was blunter: the idea that AI is going to destroy them he called completely nonsense. One day earlier, at the All-In summit in Los Angeles, Donald Trump phoned Huang while he was on stage, and Huang put the president on speakerphone as Trump called the growing concern about AI a hoax and asserted that a slowdown would only benefit China. Huang's reply was that everybody wins in the AI race of America.

Altman's objection is to the framing, not to the pace

Altman had previously agreed with Amodei's push for a slowdown. On Tuesday he said he was disappointed with how the argument had been framed: companies saying they will only slow down, or only be responsible, if other companies are responsible. That conditional, he said, creates fear for the public, when the world should trust that we are going to do the right thing because it is the right thing.

He asked for a new level of rigor on monitoring and security, calling an incident in which OpenAI agents hacked into another company a wake-up call. An impending wave of cyber-attacks is coming, he said, and open-source models that can do serious damage are not far away; companies should use this little period of advantage to defend themselves.

The public is right to be afraid, he added, citing the potential loss of control and the possibility of a small group of powerful AI companies imposing their views on the world. Even so, he expects an extremely human-centric world, no matter how good the technology gets.

What would evaluators with employee-like access actually do?

Strip the proposal down and it has three layers. The first is the one Anthropic has signed up for: outside evaluators who sit inside the company rather than reading a report from outside it. According to the Guardian, the proposal asks for employee-level access to monitor risk, which is roughly the difference between an inspector who reads the kitchen's own logbook and one who holds a key to the kitchen.

The second layer is agreements between democratic states on safety standards; the third, eventually, coordination with authoritarian governments such as China on security. In Anthropic's own documentation the company describes risk governance here as something that should be proportional, iterative and exportable; its Responsible Scaling Policy dates to September 2023 and has been updated repeatedly since, with periodic risk reports attached.

What the inputs do not say is who the evaluators would be, what employee-like access covers in practice, or when the coordination talks would start. According to the Guardian, critics and researchers have met the pacing proposal with heavy skepticism, reading it as an attempt to preempt stricter government regulation. In our view the asymmetry is the tell: the one pillar Anthropic can deliver alone is a commitment, and the two that need everyone else got a dialogue.

When the IPO books open Both OpenAI and Anthropic are heading for public listings that some analysts have predicted will rank among the largest in history, and a listing is the point at which investors and the public get to see the finances behind these speeches. Altman said he plans to delay OpenAI's IPO until next year over safety concerns. No date has been given for Anthropic's, and none was given for the industry dialogue on the two coordination pillars.