Skip to content

Benchmarks

Doc retrieval tool claims 99% fewer input tokens

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: upstash.com

Context7 ran its own benchmark against Claude Code’s built-in WebSearch and WebFetch, arguing that feeding an agent documentation slices instead of whole pages cuts what it has to read. Across every category tested, fresh input tokens dropped by roughly 99%.

The runs used the Claude Code Agent SDK on Anthropic’s defaults, Claude Opus 4.7 as the main agent and Claude Haiku 4.5 for auxiliary tasks, over five query categories, from niche libraries to questions that name no library at all.

Average cost fell 34.56%, a smaller gap than the token counts because Claude Code’s web search leans on prompt caching and routes auxiliary calls to the cheaper Haiku. Popular libraries showed the least benefit, at 27.89% off cost. The full query set and per-query results sit in Context7’s benchmark repo.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.