Claude API enforces thinking blocks and moves to Rust

Anthropic now enforces preserved thinking blocks across turns on its recent Claude models to make them harder to distill, its heads of Engineering and Product, Katelyn Lesse and Angela Jiang, said in an episode of the Latent Space podcast published October 27. If your harness compacts context by hand, the requirement can break it, and the API’s new compaction feature takes over that job while keeping thinking intact.

The interview was recorded the day after Opus 5.5 launched. It also covers the move of the Messages API from a Python monolith to Rust, with Claude writing the code, and how Claude Managed Agents keeps an agent’s harness apart from the sandbox where its code runs.

What the thinking requirement breaks

Extracting a model’s chain of thought is one of the most powerful ways to distill it, the guests said. A recently published extraction technique brought the gap into public view. Its authors disclosed it to Anthropic responsibly and it has since been patched, but the team had known about the problem and had been exploring a requirement to preserve thinking blocks across turns for some time. The newer checks also cover use spread across multiple accounts.

Anthropic sorts the cost to developers into two kinds, the guests explained: an integration change that is annoying but still lets you do everything you did before, and a change that makes something impossible, which it tries hardest to avoid. Compaction risked the second kind. Some large customers had compaction strategies that requiring preserved thinking would have broken, so Anthropic shipped a compaction feature in the API to replace the manual version. It also contacted maintainers of open source harnesses ahead of the change and told them which upgrades they would need.

Fable 5.1 is the earlier precedent. When preserved thinking first rolled out on that model, existing customers got a grace period to make the integration changes. Anthropic tracked who qualified through trust grants, its internal records of who has been given access to a specific model or program.

By the measures Anthropic can see, distillation has become significantly harder, Lesse said. The stated reason is safety: a distilled copy would run without the safeguard classifiers Anthropic runs on its own output. Jiang added that lost revenue is not the concern: “Realistically, if you distill from us, you actually pay us.”

Opus 5.5: price, retention, effort

Opus 5.5 costs less to run than its predecessor, Anthropic says, and the exact saving depends on the workload and how much of it uses cached context. Jiang described the updated cache pricing as a way to push customers toward integration changes that lower costs for everyone, with Anthropic passing those savings on. Feedback that Opus 5 wrote in a convoluted way was the other thing the launch addressed, she said.

Opus 5.5 also ships without the data retention requirements that apply to Fable, the guests said. Anthropic added retention so it could run automated and human review against misuse or distillation spread across networks of accounts, which it cannot catch while tokens are flowing, and it decides the retention policy for each model at launch. Teams doing defensive cyber work or life science research can apply to verification programs. Once their use case is approved, the online safeguards on their access are relaxed.

In Claude Code, the default effort is now medium. The guests said defaults differ for free and prosumer users, Claude Code, Cowork, Enterprise and the API, and the setting can always be changed. They also admitted that the API’s controls have become a “funny mishmash” of effort, budgets and other parameters. Part of the pitch for Managed Agents is that Anthropic works out those settings and any breaking changes for its own harness at every model launch.

Brain and hands

Claude Managed Agents is a set of APIs for defining an agent and starting sessions with it, without building the harness or the infrastructure underneath. According to the guests, its harness runs on a durable server, while the sandbox that executes code is a separate piece created when needed, so a failed sandbox does not take the session down with it. An earlier design put the whole coding agent in a sandbox behind an API. Starting it added work before the first response, and if the sandbox disappeared, the running agent went with it.

The split keeps secrets out of reach too. Credentials are held separately and attached through a proxy, and the destinations sandbox traffic can reach are restricted, so code generated inside the sandbox has no service credential to read. The sandbox breakouts people read about, the guests said, happen to models in training, boxed in with an impossible task. Production traffic adds online safeguards on top of this separation.

Latent’s hosts turn the design into questions for anyone building an agent runtime: which parts need to survive, which ones execute untrusted work, and which secrets actually need to be present where the code runs.

Dreaming, the memory feature, is built from the same parts. One managed agent inspects another’s earlier work and writes memories back to it. The guests said Anthropic will update the underlying primitive “relatively soon” so developers can build their own multi-agent strategies, such as best-of-N or verifier agents, starting from templates. Like the other roadmap items in the episode, that is a plan and has not shipped. Work on memory, dreaming and skills has slowed compared with earlier this year because more of the team’s attention has gone to safety. On evals for agents, the guests’ advice was that under-investing is the most common mistake: most teams need a fairly large suite, a simple one beats none, and Claude is good at writing evals for agents.

The Rust port

A little over a year ago, the Messages API was a Python monolith running in a single GCP region, the guests said. It has since gone multi-region, but CPU became the bottleneck. Anthropic negotiates its CPU with its cloud providers and splits it across product services, agent sandboxes, CI and research, and CI keeps growing because, in their words, “you’d be surprised how much more code is getting written.” Anthropic’s CI jobs had already grown 25x over the six months to September.

The team chose Rust even though, as one guest put it, “we don’t have a single Rust developer in the room.” The guest described the split this way: “Claude is going to write the code. The humans design the system.” Rather than migrating the API piece by piece over several years, the team built a full fork of it in Rust, ran shadow traffic through both versions, checked that the Python and Rust outputs matched, then turned the new service on gradually. The port added no features. Static analysis flagged any feature built on the Python side without its Rust counterpart, and the rollout finished the week before the interview was recorded. A couple of people with more Rust experience were hired along the way.

The guests put the CPU saving at roughly 40-50% and the improvement in time to first token at 10-20%, larger on longer-running workloads. For now those are only an executive’s ballpark estimates and have not been independently validated. The gains come from the API service layer, the code that processes requests and delivers responses around the model. Trimming that layer shortens the wait for the first token and leaves the model’s generation speed unchanged.

The team that did the port is now moving on to re-architecture, including a Messages API that would store conversation state on Anthropic’s side for a period, so callers no longer resend the whole conversation on every turn. Prompt caching still requires that resend. The guests expect the largest latency and efficiency gains on long conversations, which they said account for the majority of token volume even though few users run them.