Skip to content

Community projects

WMTrace won’t ship an Anthropic watermark detector

6 min read

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: github.com

WMTrace, an open-source watermark workbench posted to Show HN on August 15, lets developers embed and detect LLM text watermarks on their own machine, and it shows the statistics behind every result. The GitHub repository is careful about what it can’t do. It has no detector for Anthropic’s new Claude watermark, and the README says there won’t be one “until an official verification API or reproducible public specification exists.”

So the tool can’t tell you whether a paragraph came from Claude. What it can do is show how this kind of mark works, using schemes with published keys that you can switch on, take apart and test on your own text.

Why not Claude text?

Anthropic describes its watermark as a change to the source of randomness in word choice. An ordinary random number generator no longer decides which word the model picks. A key plus a few preceding words decide it instead. Anyone holding the key can check whether a word sequence matches the choices Claude would make under that key, and then assign a probability that Claude generated the text. The company says the mark relies on low-stakes choices such as “overcast” versus “grey” that come up many times in a text, that it never pushes Claude toward a word it wouldn’t have considered anyway, and that nothing is added to the text: no hidden characters. (Our full breakdown of the mechanism covers the rest.)

Tech Times reports that Anthropic has not published the specific technical method it uses. Detection is gated too. Per the Claude Help Center, it is in private preview for eligible organizations such as regulators, law enforcement, media, fact-checkers and independent researchers. The same page says the mark travels with copy-pasted text and “may persist through some editing”, that Anthropic is adding it to outputs from models released before August 2, 2026, and that the Amazon Bedrock rollout will complete by October 17.

WMTrace’s answer is a design rule: “No guessed provider secrets.” Every key in the build is a published research key. The zero-width detector, which looks for invisible Unicode payloads, has nothing to find in Claude output either, since Anthropic says its mark adds no characters.

Anthropic gives the EU AI Act as its reason: “As of August 2, the EU requires AI providers serving its market to mark AI-generated content.” It says other major model developers have signed the same Code of Practice and will implement their own watermarks. By Tech Times’ account, OpenAI’s EU compliance statement confirmed that text watermarking remains harder to deploy at scale.

Inside the green-list detector

Most of this goes back to the green-list scheme Kirchenbauer et al. at the University of Maryland described in 2023. A pseudorandom hash, seeded by the preceding tokens, splits the vocabulary into green and red, and sampling is nudged slightly toward green. On a long enough text, a statistical test picks up the skew.

WMTrace’s kgw-toy detector implements that idea with γ=0.25 and a two-word context. With γ at 0.25 a quarter of the vocabulary is green at each position, so unmarked text should land near a quarter green by chance. Detection regenerates the partition at each position, counts how many actual tokens were green, and reports N, the green count, a z-score, an exact binomial p-value and a Holm-adjusted p. Holm’s correction tightens p-values when several tests run on the same text, so running all the detectors at once doesn’t manufacture a hit.

On the built-in demo sample, the Analyze tab shows z = 16.1 and a Holm-adjusted p of 3.5e-48, along with a per-token heatmap that colours each green hit. A second keyed detector, unigram-toy, uses a fixed green list that doesn’t depend on context.

Short text is where this kind of scheme breaks down. Passages under roughly 200 tokens don’t build up enough signal for reliable p-values, and the mark degrades under heavy paraphrasing, translation or rewriting. A missing mark also doesn’t mean a human wrote the text. WMTrace handles short input by refusing to score it: detectors declare a result unsupported or insufficient before they compute anything on short, low-entropy or incompatible input, and the README calls abstention “an answer.”

Running it

The tool needs Python 3.10 or later and is licensed under Apache-2.0. From a clone, pip install -e ".[dev]" installs it (or uv pip install), and wmtrace serve starts the web UI at http://127.0.0.1:8177. Adding ?demo=1 loads and analyzes the watermarked sample, and ?theme=light or ?theme=dark forces a theme. pytest runs 41 tests, among them golden tests that reproduce every worked example in the research spec, so the docs can’t drift from the code.

The CLI covers the same ground:

  • wmtrace scan document.txt runs every detector, Holm-corrected across the run, and --json returns the full evidence bundles.
  • wmtrace detect --scheme kgw-toy document.txt tests a single declared scheme.
  • wmtrace inspect document.txt checks only Unicode and formatting: zero-width characters, bidi controls, unusual spaces, mixed scripts.
  • wmtrace embed --scheme kgw-toy --length 150 --seed 11 > marked.txt generates a reproducible watermarked sample to feed back into detect.
  • echo "Meet at the usual place." | wmtrace embed --scheme zero-width --payload Hi --ascii hides an ASCII payload in invisible characters, with U+200B for 0 and U+200C for 1.

The web UI’s Embed tab does the same round trip in one click and can reveal invisible characters inline. A lexical-codebook scheme hides bits in synonym pairs using a public codebook. Every detector except the Unicode inspector is tagged illustrative. Model-based detectors (a SynthID-Text reference, Fast-DetectGPT, Binoculars), C2PA verification and provider adapters are still on the roadmap. C2PA is the other half of Anthropic’s setup: the Help Center says Claude attaches C2PA Content Credentials to files.

Grades instead of a score

Results come as versioned evidence bundles graded E0 to E5, never as an “opaque ‘97% AI’ score.” E0 means a required key, tokenizer or configuration is missing. E1 means the input is compatible but carries too little signal. E2 marks heuristic, unkeyed or uncalibrated channels, and E3 means calibrated evidence for a declared scheme, key and configuration. E4 requires trusted key provenance that ties a result to a provider, and E5 is reserved for future cryptographic verification.

The code keeps watermark detection, generic AI-text detection, model attribution and provenance verification in separate namespaces, and their results are never averaged together. Each bundle names the text representation it analyzed (raw, NFC or NFKC), because normalization is never applied silently. The README states that “a recovered payload does not prove who embedded it” and that a detector score should never be the sole evidence for punishment or accusation. The scope statement it recommends for reports reads:

This result provides evidence compatible with watermark scheme X under configuration Y. It does not by itself prove human or machine authorship, provider identity without trusted key provenance, factual accuracy, ownership, or intent.

No detector in this build can reach E4. The keyed green-list detectors top out at E3, and the zero-width, codebook and inspector checks at E2.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.