Counting four-letter runs spots Claude Opus 5 text

A detector published by Atomic14 identifies text from Claude Opus 5 by counting four-letter character runs and feeding the proportions to a linear support vector machine, scoring 0.925 ROC AUC on four authors held out of training. Anthropic began watermarking output from its new Claude models on 2 August 2026, but that mark is statistical and readable only with the company's secret key.

At a glance

• The pipeline strips everything that is not a letter, lowercases the rest, counts all 60,211 four-character runs and normalises them, so the classifier sees no words, punctuation, capitalisation or passage length.
• Run lengths were swept from one to eight characters: 28 single-letter features already reached 0.651 ROC AUC, five-character runs peaked at 0.912, and four-character runs were chosen at 0.906.
• Unlike Anthropic's watermark, the detector needs no secret key, and the published web version runs entirely in the browser, so the text being checked never leaves the machine.

Watermarking hands detection to the vendor: only Anthropic can read its own mark, which leaves everyone downstream dependent on the company's tooling. A stylometric detector assembled from stock components is the opposite arrangement, cheap and inspectable, and it appears to work well enough on a single model to be uncomfortable for anyone relying on the assumption that machine text is hard to spot. The same experiment also exposes the limit, since the classifier tracks a writing style rather than an author.

Four-character runs led at every truncation from 100 to 1,000 letters

The sweep ran from one to eight characters. Accuracy climbed steeply to four, peaked at five with 0.912 ROC AUC over 214,949 features, then fell back to 0.863 at eight. Letter frequencies alone, 28 features, reached 0.651, well clear of the 0.5 a coin toss produces.

Four characters were chosen over five because the gain is marginal and the feature count rises from 60,211 to 214,949: there are 457,000 possible four-character runs and 11.9 million five-character ones. In a typical 2,259-letter passage 81% of the four-character runs found appear exactly once, and 89% of five-character runs do.

In a truncation test on 320 passages Claude wrote from public benchmark prompts and their 320 human source documents, four-character runs led at 0.645 for 100 letters, 0.740 at 200, 0.783 at 432 and 0.867 at 1,000. Spaced n-grams, sampling every second or third character, made accuracy much worse.

519 human passages produced 1,557 Claude rewrites

The human side is 519 passages by 26 authors, 180 from books, 179 from news and 160 from blogs, all published before 2021: Project Gutenberg books from before 1929, Blog Authorship Corpus posts from 2004 or earlier, and CC-News articles published between 2016 and 2019.

Claude Opus 5 rewrote every passage three ways through the batch API, in a plain, a direct and a formal register, preserving meaning, level of detail and approximate length, for 1,557 matched pairs. Fresh passages written from scratch on the same topics were kept out of training entirely.

Four of the 26 authors were held out untouched before training, and the remaining evaluation used cross-validation in which test authors never appeared in training. Splitting by author rather than by passage prevents the classifier from learning to recognise individual writers, which it does very well.

Held-out authors scored 0.925 against 0.906 in cross-validation

The four unseen authors scored 0.925 ROC AUC, close to the 0.906 from cross-validation, which indicates the model was not quietly identifying particular writers. Passages Claude wrote from scratch on those authors' topics, excluded from training, scored 0.977.

Against 175 posts from the author's own blog written between 2006 and 2022, the median score was 0.09 and 2% landed above 0.8. Eight passages by Claude Opus 5 scored a median of 0.85, with 63% above 0.8.

On RAID, a public benchmark for machine-generated text detection holding six million passages from 11 generators across eight subject domains, 320 prompts were answered by Claude Opus 5 and scored against the 320 human documents the prompts came from, giving 0.889 on writing produced from scratch.

Where a formal register breaks it

A version trained only on the plain and direct rewrites scored 0.153 on formal-register rewrites, rating them as more human-looking than the human passages beside them: it had learned that text which got plainer is machine-written, and was handed text that got more formal.

Adding the formal rewrites lifted fresh writing from 0.881 to 0.977 and the RAID set from 0.794 to 0.889. Whether further styles would generalise beyond that is left open, and a prompt in an untrained register remains a way around the detector.