34% to 44%: the gain came from the conversation, not the memory files

Andrew Jesson fed Claude Code (Opus 5, xhigh effort) a stream of tasks from a simulated business process plus one instruction: get better. Success on held-out tasks climbed from 34% to 44%.
Over the run the agent made 84 writes and edits, every one of them in memory files. It never ran code, never spawned subagents, never used search. The notes formed a graph with correct cross-references. But across 120 held-out tasks, the files themselves were worth +1.7 percentage points, and the confidence interval includes zero.
What actually moved the number was the conversation. Logging completed tasks along with the scores they received is worth +9.9 points, and a summary written after context compaction is worth +5.4 points at a cost of $0.14 versus $6.33 for the full conversation.
A session starting from scratch solves 41 of 120 tasks in 29.8 turns and $0.86.
Up next
Claude Code no longer limits sessions to 200 spawned subagents
Claude Code v2.1.224 lets Team and Enterprise users host sessions on their own machines or containers via claude self-hosted-runner, and drops the 200-subagent-per-session cap. Concurrency and depth limits stay. Release is dated August 7 on GitHub.
Related stories
- 34 hours of agent time, about 40 minutes on the actual research question
- Claude learned proteins, not just papers about them
- Claude Fable 5.2 hits 69% on PCB layout test, up from 25%
- GPT-6 Astra hits 66.7% on wet-lab tasks, GPT-5.6 Sol 43.8%
- Scientists use Claude least where they say it matters most
- METR clocks Anthropic benchmark gains at 1.9x old pace
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

