By the end of a long session, strict solutions drop to 0.5%

Model accuracy falls apart long before the context window fills up. On the NoLiMa benchmark, 11 of 13 models with a claimed context of 128k or more scored below half their short-prompt baseline once they hit 32k tokens. GPT-4o managed 69.7% against a 99.3% baseline.
SlopCodeBench (arXiv 2603.24755, March 2026) put 11 models to work revising their own code across 93 checkpoints. Not one agent solved a task end to end. The best single checkpoint score was 17.2%, and by the final checkpoint strict solutions had fallen to 0.5% while cost climbed 2.9x. Structural degradation showed up in 80% of runs.
The author's advice: run /clear at task boundaries and fire /compact early, while there's still headroom. In Claude Code 2.1.220, /context prints the Autocompact buffer on its own line, the space held back for the future summary.
Related stories
- Why output tokens in Claude Code cost about 5x what input does
- Claude Code sends 33k tokens before it even reads your prompt; OpenCode uses 7k
- ponytail's token savings: 10.3% off the bill, not the promised 20%
- Claude went from bug hunt to abuse report on New Year's Eve
- Opus 5.5 aced a test suite at 3.4x GPT-6 Sol's cost
- Claude hands paid users a spare limit reset until Oct 22
Comments
No comments yet. Be the first.
Join the conversation
Sign in with Google to leave a comment. Your name and avatar come from your Google profile, and the comment appears after moderation.
