Context7 cuts Claude Code input tokens by roughly 99%

Context7 stripped roughly 99% of fresh input tokens in every category of a benchmark against Claude Code's built-in WebSearch and WebFetch tools, while cost per query dropped 34.56% on average. The comparison was run and published by Upstash on its own documentation-retrieval tool.

At a glance

• The comparison ran 100 documentation queries through the Claude Code Agent SDK, splitting them into five categories, with each question answered once using Context7 tools and once using WebSearch and WebFetch.
• Averaged over the runs, Web Search consumed 38,402.84 fresh input tokens per query against Context7's 386.70, and total token usage came to 136,562.10 versus 84,278.20 for Context7.
• Savings were smallest on popular libraries, where web search finds documentation easily, at 27.89% off cost, and largest on niche libraries, where Context7 cut cost 43.12% and tool calls 55.88%.

The measurement that matters to anyone budgeting agent runs is the second one. A 99% cut in fresh input tokens reads dramatic, but Claude Code's own cost mitigations absorb most of it, and the bill moves by roughly a third. The benchmark also comes from the vendor of the tool under test, and it measures volume rather than whether the answers were right.

Fresh input tokens fell from 38,402.84 to 386.70 per query on average

Input tokens in the benchmark cover everything the model has to read that is not already cached: the query, tool definitions and tool results. Averaged across the five categories, Web Search runs put 38,402.84 of them into the context per query, against 386.70 for Context7, a 98.98% reduction.

Output tokens, which include the final response, reasoning and tool invocations, dropped 52.97%, from 3,289.93 to 1,543.88. Upstash attributes the difference to format, saying WebSearch and WebFetch likely return larger page-level artifacts while Context7 returns condensed slices, leaving the model less irrelevant material to filter.

Total token usage, which also counts cache writes and cache reads, came to 136,562.10 per query for Web Search and 84,278.20 for Context7, a 36.81% reduction. Tool calls averaged 3.47 per query for Web Search and 2.55 for Context7, a 22.77% reduction.

Niche-library queries showed the largest cost cut at 43.12%

The five categories were built around different retrieval stressors: evolving libraries, niche libraries, queries spanning several libraries, popular libraries and queries that name no library at all. Each held 20 questions, among them how extended thinking works with Opus 4.7 and how Claude Code memory files are viewed and edited.

Cost reductions ranged from 27.89% on popular libraries to 43.12% on niche ones, with unspecified-library queries at 36.94%, multi-library queries at 35.60% and evolving libraries at 29.24%. Total token reductions followed the same order, from 21.70% to 50.19% across the same categories.

In the niche category Context7 also cut tool calls by 55.88%, the largest such drop of the five. Upstash says Web Search had to work harder to locate sources in a low-signal web environment. Multi-library queries pushed Context7's tool count to 3.65, its highest of the five categories, still below Web Search's 4.00.

Prompt caching and Haiku 4.5 hold the cost gap at 34.56%

Average cost per query was $0.22 for Web Search and $0.14 for Context7. Upstash attributes the gap between a 98.98% input-token cut and a 34.56% cost cut to Claude's built-in mitigations: aggressive prompt caching and auxiliary tool calls routed to Claude Haiku 4.5 rather than Opus 4.7.

Cost figures were taken from the Claude console as reported at the time of each run. Web Search averaged 84,183.37 cache-read tokens and 10,685.96 cache writes per query, against 71,436.08 and 10,911.54 for Context7, cache writes being the only average on which Context7 came out higher.

The experiment ran through the Claude Code Agent SDK, with each query appended either by an instruction to use web search and web fetch or by an instruction to use Context7. Questions within a category ran consecutively, and a five-minute pause separated the Context7 and Web Search passes to limit caching effects. Claude Opus 4.7 drove the main agent and Claude Haiku 4.5 handled auxiliary tasks.

Answer quality benchmark still pending

This test measured tokens and dollars, not correctness. Upstash says a separate benchmark covering answer quality and safety is being prepared, including Context7's filtering of prompt-injected and malicious content that open web search can pull into an agent's context; no date has been given for it. The full query set and per-query results for both setups are in the benchmark repository.