ctok counts Claude tokens offline with no API call

ctok is an unofficial Python package that counts Claude tokens without calling the API. Give it the string “hello, world” and it returns 10 tokens for models from Claude 3 through Opus 4.6, but 15 for Opus 4.7. The project’s README on Github, in its version from August 15, says ctok installs with pip install ctok, has no runtime dependencies, needs no network access and ships under the MIT license. It is not affiliated with Anthropic.
The README names the write-up “On the biology of Claude’s tokenizer” as the research behind the model. ctok turns that research into code you can run.
For a single user message, ctok works in four steps. It normalizes the text first, which includes NFC Unicode normalization and quote folding specific to each tokenizer family. Next it rewrites the text into a stream of word, case and byte markers. Then it covers that stream with the cheapest set of vocabulary pieces it can find, falling back to raw UTF-8 bytes for anything the vocabulary doesn’t contain. Last, it adds the measured message frame. token_count(text) is just len(tokenize(text)).
The output notation shows each part: ⟨bow⟩ and ⟨eow⟩ mark word boundaries, ⟨shift⟩ and ⟨caps⟩ mark case rewrites, ⟨0xNN⟩ is a byte-fallback token, and ⟨pad⟩ is part of the single-message frame. Running ctok "hello, world" on the command line prints the marked stream and its tiling.
That tiling comes with a caveat. Claude doesn’t expose token boundaries, so tokenize() returns one valid minimum-cost tiling and makes no claim about how Anthropic actually segments the text. The reconstruction targets counts.
A version string picks the family. Strings from “3.0” up to “4.7” map to v3, the default, which covers Claude 3 through Opus 4.6. Strings from “4.7” up to “5.0” map to v4.7, covering Opus 4.7 through 4.9. Anything from “5.0” up maps to v5, for Opus 5 and Sonnet 5. v5 uses the v4.7 vocabulary with a different message frame, and “hello, world” comes back to 10 tokens on 5.0. Version components are compared as integers, so “4.10” sorts after “4.9”. Python reads the float 4.10 as 4.1, which is why ctok raises TypeError if you pass a version that isn’t a string.
The two vocabularies differ sharply in size. The v3 file holds 48,645 pieces and the v4.7 file holds 15,240, less than a third as many. Each entry either has a fixed membership witness or is one of the structural marker atoms the test suite checks. A witness records that a marked piece costs one token in a calibrated probe: witness("⟨bow⟩the⟨eow⟩", "4.7") returns {'probe': 'the', 'raw': 12, 'kind': 'raw'}. A witness doesn’t prove the encoder rewrite, and it doesn’t resolve ties between tilings of equal cost.
Accuracy was measured against recorded count_tokens responses, and both v3 and v4.7 matched exactly on every corpus listed. That covers 350,000 of 350,000 Goldfish rows across 350 languages, 1,741 of 1,741 Rosetta Code documents, and 501 of 501 UDHR texts in 501 languages, a corpus that has been in-sample since August 12. Those three corpora informed which pieces went into the vocabulary. Two sets did not: MultiPL-E, with 22 programming languages, matched 22 of 22, and a separate sample of 250 Rosetta Code documents matched 250 of 250.
v5 has no row in that table. Its deviation from the recorded counts matches v4.7’s on every gated document, and separate API tests cover its message-frame rules.
The stored measurement sets also contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961 v4.7 texts came out below the recorded count. The README notes that this does not guarantee the result for arbitrary input. The public gates can be rerun with uv run pytest and uv run python tests/gates.py.
Up next
Flaky tests after a 10x speedup, and what Claude got wrong
1360 integration tests dropped from around 40 minutes to around 4 once they ran in parallel, and the new flakiness took two passes with Claude to clear: the first round of fixes sounded plausible and did nothing.
Related stories
- Reverse-engineering Claude’s tokenizer: words live between ^ and $
- claude-api-guard fails CI on broken Claude API calls
- Linear and Ramp users can pay for Claude without an API key
- Tokendoctor cut its author’s Claude Code cost 38%
- Claude Desktop on Linux freezes over a 27-byte reply
- memsaga gives Claude Code’s memory files a git history
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

