Claude went from bug hunt to abuse report on New Year’s Eve

Alex Palcuie, who is on call for Claude at Anthropic, has moved his answer to “Can Claude fix itself?” from March’s “no” to “maybe”, and he still wants a human to approve every fix in production, per a write-up of his talk at Signals Berlin earlier in September, published on Sylvainkalache on September 25. If your team puts Claude on incidents, his scorecard puts the risk in the diagnosis: Claude investigates faster than any human responder, but making sense of what it finds is uneven, and in one of the three incidents he described it settled on a convincing, wrong cause.

Palcuie grades Claude on the OODA loop, borrowed from the U.S. Air Force: observe, orient, decide, act. Observing is where LLMs shine, searching logs, querying metrics and chasing several leads in parallel. Deciding is in good shape too, with Claude proposing fixes, comparing their trade-offs and testing them against past incidents. Acting is where the line is still to be drawn: beyond the human sign-off, Palcuie wants to understand every command he runs. Anthropic’s published setup for Claude Tag, its CI team’s on-call agent, draws the same line: humans deploy every fix.

Three incidents

On New Year’s Eve, Opus started returning more HTTP 500 errors, and Palcuie asked Claude Code to investigate. It traced them to a bug in image preprocessing: the failing requests contained exactly 22 images. At that point, Palcuie says, he would probably have just filed a ticket for the API team. Claude went on to check who was sending them and found 200 accounts sending similar requests at the same time, all from a batch of 4,000 created three days earlier with the same email pattern. It recommended bringing in the safeguards team, which confirmed the abuse. He calls it his Move 37 moment, after the AlphaGo move against Lee Sedol that experts first took for a mistake.

The second came when the inference router drained a healthy cluster and overloaded the servers that took its traffic. While the humans in the channel blamed the data center, Claude noticed a handful of servers reporting more tokens per second than is physically possible and concluded that the system was fine and the measurement was wrong. The metric collectors had just been resharded, and for five minutes two shards scraped the same servers, inflating the rate the router relied on. Claude proposed and tested a fix. For Palcuie it was another Move 37: Claude had gone into the observability stack, a system he is not an expert in himself.

In the third, one cluster returned errors for 41% of requests over five minutes, and Claude noticed that local cache hit rates had dropped just beforehand. The timing made a lost cache look like the cause. An hour later, another engineer’s Claude session found what was missing: requests that missed the local cache still retrieved data from a shared cache, the overall hit rate had barely dipped, and the overloaded servers were the problem. Palcuie says he could have made the same mistake, because the graph fit the explanation he expected. He recalled a warning from Steve McGhee, his first manager at Google: looking at graphs without an explicit reason can turn up problems that aren’t really problems.

Starting further back

The write-up’s author works at Rootly, which builds AI-assisted incident response tools, and argues that most teams start from further back than Palcuie, who reached Staff level in eight years at Google and now has Anthropic’s best models and almost unlimited tokens. The reason has nothing to do with the model: an agent is only as good as the context it can access. Code and infrastructure may lack instrumentation, security restrictions may block access to data that exists, and key context may live only in people’s heads.

Cost cuts the same way. Engineers have repeatedly told the author that sending application logs to Datadog cost far more than their budget allowed, leaving the agent without the context it needed.

A classifier for production

For more autonomy than a human sign-off on each fix, Palcuie argues, a team would need a production action classifier that approves routine operations and flags riskier ones for review. Anthropic’s Claude blog describes a classifier of that kind in Claude Code’s auto mode: it reviews each tool call before it runs, checking for destructive actions such as mass file deletion, sensitive data exfiltration or malicious code execution. Safe actions proceed and risky ones are blocked; if Claude keeps insisting on a blocked action, it eventually triggers a permission prompt to the user.