Reverse-engineering Claude’s tokenizer: words live between ^ and $

An independent researcher has reconstructed how Claude’s tokenizer splits text and described the results in a Substack post published August 13. His main finding is that Claude doesn’t use the standard byte-level BPE that most other labs ship. It appears to wrap words in boundary markers and to encode capital letters as a separate signal.
The researcher started looking because of cost. “Claude’s tokenizer remains a mystery, and with the recent 40%+ increase in token cost for basic English text, even more so,” the post opens. Anthropic gives a lower figure. Developers Digest quotes Anthropic’s pricing page as saying that Opus 4.7 and later “may use up to 35% more tokens for the same fixed text.” The same outlet cites the models overview, where 1M tokens comes to roughly 555k words on Opus 4.8 and 4.7 and roughly 750k words on Opus 4.6 and Sonnet 4.6. Per-token pricing for Opus 4.8 stayed at the Opus 4.6 level: $5 per million input tokens and $25 per million output tokens.
The method relies on a property of pairwise BPE that can be tested. BPE builds its vocabulary by merging pairs of existing pieces, so every token that isn’t a base byte is made of two smaller tokens, and any one-token span has to split somewhere into two parts that are both in the vocabulary. The Hangul syllable 최 (choi), a very common surname, breaks this rule. It is a single token, yet every Korean character that shares its 2-byte UTF-8 prefix or suffix costs 3 tokens, one per byte, so neither half is a token anywhere else. The author says the rule that seems to explain the counts consistently is minimum-piece tokenization, the approach used in MinGram and PathPiece: cover the text with the fewest vocabulary entries possible.
Words between ^ and $
The author calls boundary markers one of the main novelties. He writes them as ^ and $, inferred begin-of-word and end-of-word tokens, and word-like spans are wrapped in them: “tokenizers” becomes [^token][izers$]. The markers explain some odd counts. On their own, “semiconduct” tokenizes as [^semiconduct][$], “usercontent” as [^][usercontent$] and “romagnet” as [^][romagnet][$]. Each fragment is a vocabulary entry, but only in the right position. Glue them together as “semiconductromagnetusercontent” and the result is the expected three tokens. “Telecommunications” is one token, while “telecommunicationsy” breaks into four: [^telecommun][ic][ation][sy$]. This corrects the author’s previous post, which concluded that most Claude tokens were whole words. What that post actually measured was whether the fully marked ^word$ form was a single vocabulary entry.
The markers seem to earn their place through spaces. The author thinks any space between $ and ^ is removed before encoding and that $^ is turned back into a space on decoding. If so, many spaces in code and prose cost nothing. The model also doesn’t have to split its training data between a space-prefixed version of a word and a bare one. Punctuation runs get markers as well, but only on the side that borders a space: in a== b the operator becomes ==$.
Shift and caps codes
Capitalization works on the same principle. The tokenizer doesn’t keep separate entries for nasa, Nasa and NASA. Title case appears to get a shift marker (↑) and all caps a caps marker (⇪), each followed by the lowercase form. Like ^ and $, a marker can be part of a token or a token of its own. It applies only to a whole pretoken, so GaN and WiFi stay literal. MERCHANTABILITY is a single token, [⇪^merchantability$]. Add a lowercase “q” and both the caps code and the word marking break, which leaves [^M][ER][C][H][A][N][T][A][B][I][L][I][T][Y][q$]. The v4.7+ tokenizer simplifies this considerably and drops the caps-lock code.
The rest of the reconstruction is more conventional, with a few oddities:
- Han, Korean, emoji and many rare scripts get no word markers and are always tokenized one character at a time.
- A character with no dedicated token falls back to UTF-8 bytes, but only within that character and strictly by prefix, similar to what SCRIPT-BPE proposes.
- Digits are grouped in threes.
- Text is normalized to NFC with some character replacements, notably curly quotes in the earlier version.
- Runs of spaces, newlines and tabs have their own tokens. Runs of 1 through 30 newlines are tokens, as are runs of 32, 33, 40, 48, 64, 98 and 128. A run of 31 is not.
There is also a fixed overhead of 7-8 tokens: “a” costs 8, and a single digit costs 9. The author says closer investigation suggests this comes from the chat formatting around the text, which begins with ^ and ends in two newlines, along with some other unknown content. The leading ^ can produce puzzling results when you’re trying to work out what is and isn’t a token.
A smaller vocabulary
Based on the tokens found so far, the author estimates the vocabulary at around 49,000 to 55,000 entries for Claude v3 through v4.6, and only 16,000 to 20,000 for later versions. Developers Digest reports that the newer tokenizer is used by Opus 4.7, Opus 4.8, Fable 5 and Mythos 5. The earlier one covers everything before Opus 4.7, including Opus 4.6, Sonnet 4.6 and Haiku 4.5.
The author acknowledges the limits: “All of the above is somewhat speculative, and none of it is from official sources.” Some behaviour remains unexplained in languages that use many markers. In Devanagari, the predicted counts can be off by as much as 10%. Most of the work went into the older tokenizer, and the errors for v4.7+ are slightly larger. The reconstruction code is on GitHub.
Up next
DeepSeek open-sources its agent harness under MIT
DeepSeek open-sourced DeepSeek Harness, a Node.js agent runtime, under the MIT license, and it picked up more than 33,000 GitHub stars in a few hours. Every piece is swappable: model adapter, tool registry, session log, even the agent loop itself.
Related stories
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

