ctok counts Claude tokens offline with no API call

An unofficial Python package called ctok, installed with pip install ctok, reconstructs Claude token counts offline without an API call, network access or runtime dependencies, matching recorded count_tokens responses exactly on 350,000 Goldfish rows across 350 languages and shipping on Github under MIT.

At a glance

• The package normalizes text with NFC and family-specific quote folding, rewrites it into a stream with word, case and byte markers, then finds a minimum-cost tiling over the measured vocabulary and UTF-8 byte fallback.
• The two vocabulary files contain 48,645 v3 pieces and 15,240 v4.7 pieces, and the notation in the output marks word boundaries, case rewrites, byte-fallback tokens and the single-message frame.
• A version string selects the family: v3 by default for Claude 3 through Opus 4.6, v4.7 for Opus 4.7 through 4.9, and v5 for Opus 5 and Sonnet 5 on the v4.7 vocabulary.

Offline counting matters for anyone budgeting context in tooling, because a local count removes a network round trip from prompt assembly, cache planning and cost estimation, and it still works where the API is unreachable. The caveat is that the model is measured rather than official, so it likely tracks Anthropic's counter only as far as the recorded responses reach, and any change to a vocabulary or a message frame would surface as silent drift.

A version string selects the family, and the same text costs 10 tokens on v3 and 15 on v4.7

The family is selected with a version string. token_count("hello, world") returns 10 on the default v3 family, 15 when called with "4.7", and 10 with "5.0". The command-line entry point ctok "hello, world" prints the marked stream together with its tiling.

Version components are compared as integers, so "4.10" sorts after "4.9". Python reads the float literal 4.10 as 4.1, so passing a non-string version raises TypeError. Versions from "3.0" up to "4.7" select v3, the default family, which covers Claude 3 through Opus 4.6.

The version argument is a string such as "4.7". Versions from "4.7" up to "5.0" select v4.7, which covers Opus 4.7 through 4.9. Versions of "5.0" or higher select v5 for Opus 5 and Sonnet 5, and v5 reuses the v4.7 vocabulary with a different message frame.

Held-out MultiPL-E and a separate 250-document Rosetta Code sample matched recorded counts exactly

The published results compare ctok with recorded count_tokens responses. Goldfish, 350 languages and 350,000 rows, and Rosetta Code, 1,741 documents, were used for mining and were exact for v3 and v4.7. UDHR, 501 languages, has been in-sample since 2026-08-12 and was exact for both.

Two sets did not inform piece selection: MultiPL-E, 22 programming languages, and a separate Rosetta Code sample of 250 documents. Both matched recorded counts exactly for v3 and v4.7, and Goldfish, the main Rosetta sample and UDHR are the corpora that informed piece selection.

The v5 family is omitted from the accuracy table because its deviation from recorded counts matches v4.7's on every gated document, with separate API tests covering its message-frame rules. The stored measurement sets contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961 v4.7 texts.

The 15,240 v4.7 pieces are exposed through the pieces and witness calls

pieces("4.7") returns 15,240 entries in the v4.7 vocabulary, and witness("⟨bow⟩the⟨eow⟩", "4.7") returns {'probe': 'the', 'raw': 12, 'kind': 'raw'}. A witness records that one marked piece costs one token in a calibrated probe. It does not prove the encoder rewrite or resolve ties between equal-cost tilings.

The output notation makes the internal structure visible: ⟨bow⟩ and ⟨eow⟩ mark word boundaries, ⟨shift⟩ and ⟨caps⟩ mark case rewrites, ⟨0xNN⟩ marks a byte-fallback token, and ⟨pad⟩ is part of the single-message frame. Because Claude does not expose token boundaries, tokenize returns one valid minimum-cost tiling rather than a claim about Anthropic's exact segmentation.

The package is unofficial and not affiliated with Anthropic, and the research behind it is described in the write-up On the biology of Claude's tokenizer. For one user message ctok adds the measured message frame after tiling, and token_count(text) is defined as len(tokenize(text)).

Reproducing the published gates

The public gates run with uv run pytest and uv run python tests/gates.py, and tests/test_witness.py checks every published witness while requiring complete witnessed-or-special coverage, with every vocabulary entry carrying a fixed membership witness or counting as a structural marker atom. The project states that the absence of under-counts in the stored measurement sets does not guarantee the result for arbitrary input.