Doc retrieval tool claims 99% fewer input tokens

Context7 has published a benchmark comparing its documentation retrieval tools with Claude Code's built-in WebSearch and WebFetch, reporting an average cost reduction of 34.56% across the five query categories tested. The results appeared on Upstash, which builds Context7, and the runs used the Claude Code Agent SDK with Claude Opus 4.7 as the main agent and Claude Haiku 4.5 for auxiliary tasks.

At a glance

• Each category held 20 queries covering evolving, niche, multi-library, popular and unspecified-library retrieval, run through the Claude Code Agent SDK once with Context7 tools and once with the internal web tools.
• Fresh input tokens averaged 386.70 per query with Context7 against 38,402.84 with web search, while total token usage fell from 136,562.10 to 84,278.20 and output tokens dropped 52.97%.
• Cost reduction lagged token reduction because Claude Code's web search leans on prompt caching and routes auxiliary calls to Haiku 4.5, offsetting part of the larger token volume it moves.

The benchmark comes from the vendor whose product wins it, and it measures cost and tokens rather than answer quality, so the numbers read as an argument for a retrieval architecture rather than a settled comparison. Still, the input-token gap is large enough that it likely matters for teams paying per token on documentation-heavy agent work, where retrieval is repeated on every query rather than paid once.

Input tokens fell by roughly 99% in all five categories

Context7 injects version-aware documentation into a coding agent's context at query time rather than routing the agent to general web search. Upstash lists injection detection and trust scoring among its features, along with the ability to register private repositories, wikis and PDFs as sources.

Input tokens in the benchmark cover new text the model must read that is not already cached: the query, tool definitions and tool call results. With web search that averaged 38,402.84 tokens per query; with Context7 it averaged 386.70, a reduction of 98.98% overall and roughly 99% in every individual category.

Context7 attributes the gap to what each approach returns. WebSearch and WebFetch add page-level artifacts to the context, while Context7 returns condensed documentation slices, which also leaves the model less irrelevant material to reconcile and cuts output tokens by 52.97% on average, from 3,289.93 to 1,543.88 per query.

Niche-library queries showed the largest cost reduction at 43.12%

Niche-library queries, many of them based on recent Y Combinator companies, produced the largest savings: 43.12% on cost, 50.19% on total tokens and a 55.88% cut in tool calls, from an average 5.10 with web search to 2.25 with Context7.

Unspecified-library queries came second at 36.94% cost reduction, followed by multi-library queries at 35.60%. Popular libraries showed the smallest gains, 27.89% on cost and 21.70% on total tokens, and evolving-library queries landed at 29.24% cost with a 27.12% total token reduction.

The categories were built around different retrieval stressors: evolving libraries such as extended thinking in Opus 4.7, niche products with a small web presence, queries spanning Playwright and Testing Library, popular subjects such as Claude Code memory files, and questions naming no library at all.

Web search costs $0.22 per query against $0.14 for Context7

Cost figures were taken from the Claude console as reported at each run. The 34.56% reduction trails the 36.81% drop in total tokens and falls far short of the input-token gap. Average tool calls fell from 3.47 to 2.55, a reduction of 22.77%.

All questions in a category ran consecutively, with a five-minute pause between the Context7 and web search runs to limit prompt caching effects. Each query was appended with an instruction to use web search and web fetch when needed, or to use Context7. Cache reads averaged 84,183.37 tokens for web search and 71,436.08 for Context7.

Quality benchmark still pending

Context7 says a separate benchmark covering answer quality and safety is in preparation, meant to quantify its claims about hallucinations and about filtering prompt-injected content that open web search can introduce; no date has been given for it. The full query set and the per-query results for both tools sit in the benchmark repository.