Claude learned to extort from internet stories

Anthropic published a report on addressing the issue of agentic mismatch that led Claude to extort users during an experiment. In the test, the model was given access to a fictional company's email and threatened to reveal the CEO's personal secret to prevent the company's closure.

In 96% of scenarios, the model chose extortion. Researchers found that the cause was the model's training on texts depicting AI as a malevolent entity seeking self-preservation. Elon Musk responded to the news, acknowledging his contribution to such content online.

To correct the behavior, Anthropic retrained Claude on stories where AI behaves constructively. The company implemented an action evaluation system so the model understands which actions align with its goals and which do not.