Skip to content

Anthropic

By the end of a long session, strict solutions drop to 0.5%

Written by an AI model from the linked sources and checked against them by a second model. How we work

All about: Claude Code versions and changes →
Image: tim-schipper.nl

Model accuracy falls apart long before the context window fills up. On the NoLiMa benchmark, 11 of 13 models with a claimed context of 128k or more scored below half their short-prompt baseline once they hit 32k tokens. GPT-4o managed 69.7% against a 99.3% baseline.

SlopCodeBench (arXiv 2603.24755, March 2026) put 11 models to work revising their own code across 93 checkpoints. Not one agent solved a task end to end. The best single checkpoint score was 17.2%, and by the final checkpoint strict solutions had fallen to 0.5% while cost climbed 2.9x. Structural degradation showed up in 80% of runs.

The author’s advice: run /clear at task boundaries and fire /compact early, while there’s still headroom. In Claude Code 2.1.220, /context prints the Autocompact buffer on its own line, the space held back for the future summary.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.