Reverse-engineering Claude's tokenizer: words live between ^ and $

An independent reconstruction of Claude's tokenizer finds that token counts are best explained by minimum-piece segmentation of the kind used in MinGram and PathPiece, with the author's best estimate of the vocabulary at 49,000 to 55,000 entries for versions v3 through v4.6.

The analysis, published on Substack, puts later versions at roughly 16,000 to 20,000 entries and was triggered by the recent increase of more than 40% in token cost for plain English text. Its author describes the whole reconstruction as speculative and drawn from no official sources.

At a glance

• A pairwise BPE construction would require every non-base token to be built from two smaller vocabulary pieces, and Korean syllable tests show Claude's counts do not follow that constraint.
• Word-like spans are wrapped in inferred begin and end markers, written as ^ and $, which is why telecommunications costs one token while telecommunicationsy costs four in the same reconstruction.
• Capitalisation is encoded as a marker plus a lowercased form applied only to whole pretokens, so MERCHANTABILITY is one token while adding a lowercase q pushes the same string to fifteen.

Why it matters Token accounting drives cost and context budgeting, and a scheme this far from byte-level BPE means intuitions carried over from other tokenizers likely misprice Claude workloads. Marker-based boundaries appear to make much of the whitespace in code close to free, while mixed-case identifiers and stray suffixes fall off that path and fragment into single characters. The gap between a one-token word and a fifteen-token one is a budgeting problem for anyone estimating API spend.

Korean syllables that share bytes with 최 each cost three tokens

A pairwise BPE construction requires every non-base token to be built from two smaller vocabulary pieces, which makes the scheme testable. The Hangul syllable 최 is a single token, yet other Korean syllables sharing its two-byte prefix or suffix cost three tokens, their length in bytes.

Han, Korean, emoji and many rare scripts carry no word markers and are tokenized character by character. Characters without a dedicated token fall back to UTF-8 bytes, strictly prefix-based and only within a character. Normalization is NFC, with some character replacements, notably curly quotes in the earlier version.

Measurements also show a fixed overhead of seven to eight tokens, so a lone letter a costs eight and a single digit costs nine. The author attributes this to the surrounding chat formatting, which begins with ^ and ends in two newlines alongside other unknown content.

Concatenating semiconduct, romagnet and usercontent yields exactly three tokens

Word-like spans are wrapped in inferred begin and end markers, rendered as ^ and $, so tokenizers splits as [^token][izers$]. In isolation semiconduct and usercontent cost two tokens each and romagnet costs three, since the marked forms are [^semiconduct][$], [^][usercontent$] and [^][romagnet][$]. Concatenated as semiconductromagnetusercontent, the string comes out at the expected three tokens.

Telecommunications is a single token, while telecommunicationsy costs four. Spans of punctuation take markers only on the side that borders a space: the operator in a== b becomes ==$. The author suggests any space in the sequence $ followed by ^ is dropped before encoding and restored during decoding, so much of the whitespace in code and prose costs nothing.

Whitespace runs have their own entries: tokens exist for 1 through 30 newlines and for 32, 33, 40, 48, 64, 98 and 128, but not for 31. Spans of spaces and tabs also have dedicated tokens, and digits are grouped in threes.

The v4.7 tokenizer drops the caps-lock code used in earlier versions

Capitalisation uses a marker plus a lowercased form: Token is encoded as a shift marker followed by token, and NASA as a caps marker followed by nasa. The marker can sit inside a token or stand alone, and it applies to a whole pretoken only, so GaN and WiFi are stored literally rather than as marked forms.

The effect shows in MERCHANTABILITY, a single token in the form [⇪^merchantability$]. Appending a lowercase q breaks both the caps code and the word marking, and the string becomes [^M][ER][C][H][A][N][T][A][B][I][L][I][T][Y][q$]. According to the write-up, the v4.7 and later tokenizer removes the caps-lock code.

What's next Unexplained behaviour remains in languages with many markers, where Devanagari counts are off by as much as 10%, and errors on v4.7 and later run somewhat larger because most of the work targeted the older version. The reconstruction code is published on GitHub, which leaves the estimates open to independent testing.