Skip to content

anthropic

By the end of a long session, strict solutions drop to 0.5%

Claude News

Model accuracy falls apart long before the context window fills up. On the NoLiMa benchmark, 11 of 13 models with a claimed context of 128k or more scored below half their short-prompt baseline once they hit 32k tokens. GPT-4o managed 69.7% against a 99.3% baseline.

SlopCodeBench (arXiv 2603.24755, March 2026) put 11 models to work revising their own code across 93 checkpoints. Not one agent solved a task end to end. The best single checkpoint score was 17.2%, and by the final checkpoint strict solutions had fallen to 0.5% while cost climbed 2.9x. Structural degradation showed up in 80% of runs.

The author's advice: run /clear at task boundaries and fire /compact early, while there's still headroom. In Claude Code 2.1.220, /context prints the Autocompact buffer on its own line, the space held back for the future summary.

Related stories

  1. Why output tokens in Claude Code cost about 5x what input does
  2. Claude Code sends 33k tokens before it even reads your prompt; OpenCode uses 7k
  3. ponytail's token savings: 10.3% off the bill, not the promised 20%
  4. Claude went from bug hunt to abuse report on New Year's Eve
  5. Opus 5.5 aced a test suite at 3.4x GPT-6 Sol's cost
  6. Claude hands paid users a spare limit reset until Oct 22

Comments

No comments yet. Be the first.

Join the conversation

Sign in with Google to leave a comment. Your name and avatar come from your Google profile, and the comment appears after moderation.

We only use your name and avatar from Google. We never store your email address.