Skip to content

Anthropic

Counting four-letter runs spots Claude Opus 5 text

9 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: atomic14.com

A detector that counts four-character runs of letters, and looks at nothing else, separated Claude Opus 5’s writing from human writing with a ROC AUC of 0.925 on four authors it never saw in training, according to a write-up published on Atomic14 on August 18. ROC AUC asks how often a random AI passage scores higher than a random human one: 0.5 is a coin toss, 1.0 means it never gets a pair the wrong way round.

Anthropic began watermarking text from its new Claude models on August 2. The mark is statistical: a secret key and a few preceding words settle which word the model picks when several would do, and Anthropic says in its announcement that nothing is added to the text. Only the key holder can check it, and the official detector is in private preview for organisations eligible under EU law, among them regulators, media, fact-checkers and independent researchers, per Anthropic’s help center. The Atomic14 detector needs no key. It runs in a browser, and the text never leaves the machine.

From letters to a score

The method comes from questioned document examination, the field that tries to work out who wrote a disputed text, which the author looked into during a PhD. Its premise is that an author can often be identified by counting short runs of characters while ignoring words and sentence structure. Those runs are n-grams. Strip “the quick brown fox” of spaces, punctuation and capitals, slide a four-character window along thequickbrownfox one position at a time, and you get theq, hequ, equi and so on. Do that over a whole passage and it becomes a table of counts. The passage itself is thrown away; the table is all the detector ever sees.

The finished pipeline is short. Strip everything that is not a letter, spaces included, and lowercase the rest. Count every 4-gram, so each passage becomes a list of 60,211 numbers. Normalise them into proportions, feed them to a linear support vector machine that learns which counts run higher in Claude’s writing, and convert the output into a score between 0 and 1. Both the counting and the classifier are stock scikit-learn components. The detector never sees punctuation, sentence length, paragraph structure, capitalisation or any whole word, and normalisation hides how long the passage was.

“The hard part is getting data, the actual “science” part is easy,” the author writes. Human text had to be definitely human, so everything predates 2021: Project Gutenberg books from before 1929, Blog Authorship Corpus posts from 2004 or earlier, and CC-News articles from 2016 to 2019. That came to 519 passages by 26 authors: 180 from books, 179 from news and 160 from blogs. Claude Opus 5 then rewrote each passage, keeping its meaning, level of detail and rough length, under three instructions: plain (“Rewrite the following passage in clear natural English”), direct (“plain, direct modern English”) and formal (“a formal, precise register”). The result was 1,557 rewrites generated through the batch API, matched pairs in which subject, facts and length are held close and only the writing differs. Claude also wrote fresh passages from scratch on the same topics, which were kept out of all training and saved for testing.

The split matters as much as the features. Four of the 26 authors were set aside before anything was trained, and the rest were cross-validated so that the authors used for testing were never the ones used for training. Split by passage instead, the author notes, and the classifier learns to recognise individual writers, which it does very well but which is not the task.

Why four letters

The author swept the run length from 1 to 8 with everything else fixed. Single letters, just 28 features, already reach 0.651. Accuracy climbs steeply after that: 0.757 for pairs (673 features), 0.857 for three-letter runs (8,590), 0.906 for four (60,211), and a peak of 0.912 for five (214,949). Then it falls: 0.907 at six, 0.890 at seven and 0.863 at eight letters with 843,596 features. In the author’s reading, longer runs carry more information until there is too little data to support them. Spaced n-grams, built from every second or third character, made accuracy much worse.

The final detector uses four letters, not the peak. Moving to five adds 0.006 of AUC for roughly three and a half times as many features, and the gain shows up only on long passages. The reason is sparsity. There are about 457,000 possible 4-grams and 11.9 million possible 5-grams. In a typical corpus passage of 2,259 letters, the author found 1,750 distinct 4-grams, 81% of them appearing exactly once, and 1,991 distinct 5-grams, 89% appearing once. At 200 letters, 90% of 3-grams and 97% of 5-grams appear only once.

Short text is where the detector struggles most. On 320 passages Claude wrote from prompts in the RAID benchmark, scored against their 320 human source documents and truncated to different lengths, the 4-gram version beat the 2-, 3- and 5-gram versions at every length. Even so, it reached only 0.645 at 100 letters and 0.740 at 200, then 0.783 at 432 letters and 0.867 at 1,000. The 5-gram version came within a thousandth of it at 1,000 letters, with 0.866.

Tests outside the training data

The four held-out authors were scored once, at the end, and came in at 0.925, against 0.906 from cross-validation. For the author, the closeness of those two numbers was the main thing to check, since it suggests the classifier was not quietly learning to recognise particular writers.

The author’s own blog made a second test: 175 posts written between 2006 and 2022 got a median score of 0.09, and 2% of them scored above 0.8. Eight passages from Claude Opus 5 got a median of 0.85, with 63% above 0.8. The human side of that test is large; the Claude side is only eight passages.

RAID was the harder check. It is a public benchmark for machine-generated text detection with six million passages from 11 generators across eight subject domains. The author took 320 of its prompts, had Claude Opus 5 answer them, and scored the answers against the 320 human documents the prompts came from. It was a different corpus and a different task, since these were written from scratch rather than rewritten, and the detector scored 0.889. The fresh passages written on the four unseen authors’ topics, never used in training, scored 0.977.

The author’s summary is that counting four-letter sequences, “with no understanding of the text whatsoever”, separates human writing from Claude’s rewriting of that same writing about 93% of the time on authors it has never read. “I did not expect it to work that well.”

What Claude does to letters

The model is simple enough to inspect. For the plain rewrites, at three characters so the fragments stay readable, the strongest features pushing towards Claude include -ing endings, -ly adverbs (ely, tly, and ctl from words like “exactly”), “that”, “though”, “through” and Ive, the contraction “I’ve” with its apostrophe stripped. On the human side sit “which”, “not”, “but”, “will”, “was”, the -tion and -ation endings, “of the” and “of a” (fth, oft, ofa), and Iam and Iwi: “I am” and “I will” written out in full.

Comparing each rewrite with the exact passage it came from isolates the writing from subject and era. Under the plain instruction, Claude’s versions use more g, h, l, r, n and y, and less i, s, o, f, m and c. Under the direct instruction, they use more g, h, w, k, l and y, and less i, c, f, e, s and m. According to Claude’s own reading of the data, which the author reports, i, c, f, s and m are the letters of words English borrowed from Latin and French, such as “consideration” and “specific”. G, h, w, k and y belong to the older Germanic core: “think”, “work”, “know”. A word-level check backs this up. Latin endings drop under both instructions, and in the plain rewrites mean word length barely moves, so the shift comes from which words Claude chose, not from how long they were.

The formal instruction flips it. Claude’s formal rewrites use more c, e, r, i, n and p, and less h, w, k, l, o and g. Every letter has swapped sides. At the word level, both Latin endings and mean word length go up.

The formal blind spot

To see how far the detector generalises, the author trained it twice: once on all three instructions, and once on the plain and direct rewrites only. With formal rewrites left out of training, it scored 0.153 on them, reliably rating Claude’s formal text as more human-looking than the human passages beside it. Trained on all three, it scored 0.977 on the same set. “It is not confused and it is not broken,” the author writes. It had learned that text that got plainer is AI, and it was then handed text that got more formal.

  • Plain rewrites: 0.827 trained on plain and direct, 0.899 trained on all three
  • Direct rewrites: 0.879 and 0.900
  • Formal rewrites: 0.153 and 0.977
  • Fresh writing, in neither training set: 0.881 and 0.977
  • RAID passages, outside the corpus: 0.794 and 0.889
  • The author’s blog, 8 AI passages: 0.986 and 0.981

Adding a third style did more than teach the detector that style. The fresh passages were in neither training set, yet they rose from 0.881 to 0.977, and RAID rose from 0.794 to 0.889. The author thinks more styles would probably make the detector more general still. The same table shows the weakness too: the author acknowledges that the detector can be beaten by asking Claude for a style it has not been trained on.

The write-up itself was a test case. Fed into the detector, the post comes out as written by Claude, even though the author edited it substantially.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.