Skip to content

Developer tools

ctok counts Claude tokens offline with no API call

3 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: github.com

ctok is an unofficial Python package that counts Claude tokens without calling the API. Give it the string “hello, world” and it returns 10 tokens for models from Claude 3 through Opus 4.6, but 15 for Opus 4.7. The project’s README on Github, in its version from August 15, says ctok installs with pip install ctok, has no runtime dependencies, needs no network access and ships under the MIT license. It is not affiliated with Anthropic.

The README names the write-up “On the biology of Claude’s tokenizer” as the research behind the model. ctok turns that research into code you can run.

For a single user message, ctok works in four steps. It normalizes the text first, which includes NFC Unicode normalization and quote folding specific to each tokenizer family. Next it rewrites the text into a stream of word, case and byte markers. Then it covers that stream with the cheapest set of vocabulary pieces it can find, falling back to raw UTF-8 bytes for anything the vocabulary doesn’t contain. Last, it adds the measured message frame. token_count(text) is just len(tokenize(text)).

The output notation shows each part: ⟨bow⟩ and ⟨eow⟩ mark word boundaries, ⟨shift⟩ and ⟨caps⟩ mark case rewrites, ⟨0xNN⟩ is a byte-fallback token, and ⟨pad⟩ is part of the single-message frame. Running ctok "hello, world" on the command line prints the marked stream and its tiling.

That tiling comes with a caveat. Claude doesn’t expose token boundaries, so tokenize() returns one valid minimum-cost tiling and makes no claim about how Anthropic actually segments the text. The reconstruction targets counts.

A version string picks the family. Strings from “3.0” up to “4.7” map to v3, the default, which covers Claude 3 through Opus 4.6. Strings from “4.7” up to “5.0” map to v4.7, covering Opus 4.7 through 4.9. Anything from “5.0” up maps to v5, for Opus 5 and Sonnet 5. v5 uses the v4.7 vocabulary with a different message frame, and “hello, world” comes back to 10 tokens on 5.0. Version components are compared as integers, so “4.10” sorts after “4.9”. Python reads the float 4.10 as 4.1, which is why ctok raises TypeError if you pass a version that isn’t a string.

The two vocabularies differ sharply in size. The v3 file holds 48,645 pieces and the v4.7 file holds 15,240, less than a third as many. Each entry either has a fixed membership witness or is one of the structural marker atoms the test suite checks. A witness records that a marked piece costs one token in a calibrated probe: witness("⟨bow⟩the⟨eow⟩", "4.7") returns {'probe': 'the', 'raw': 12, 'kind': 'raw'}. A witness doesn’t prove the encoder rewrite, and it doesn’t resolve ties between tilings of equal cost.

Accuracy was measured against recorded count_tokens responses, and both v3 and v4.7 matched exactly on every corpus listed. That covers 350,000 of 350,000 Goldfish rows across 350 languages, 1,741 of 1,741 Rosetta Code documents, and 501 of 501 UDHR texts in 501 languages, a corpus that has been in-sample since August 12. Those three corpora informed which pieces went into the vocabulary. Two sets did not: MultiPL-E, with 22 programming languages, matched 22 of 22, and a separate sample of 250 Rosetta Code documents matched 250 of 250.

v5 has no row in that table. Its deviation from the recorded counts matches v4.7’s on every gated document, and separate API tests cover its message-frame rules.

The stored measurement sets also contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961 v4.7 texts came out below the recorded count. The README notes that this does not guarantee the result for arbitrary input. The public gates can be rerun with uv run pytest and uv run python tests/gates.py.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.