Anthropic joins a shared robots.txt opt-out for AI training

OpenAI said on October 30 that it will honor a shared robots.txt signal for generative AI training. Anthropic, Google and Microsoft back the same token, __usage_ai-training, which a site puts on the User-Agent line of an ordinary group. Under the draft that defines it, a wildcard User-agent: * group never counts as the token, so no robots.txt published today expresses a training preference through this mechanism, full blocks included, until the token group is added.
The group uses ordinary robots.txt syntax. The draft’s own example is User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/: no training use anywhere except the press kit. Per-crawler rules stay in force alongside it. OpenAI says the new signal “will not replace” its existing robots.txt controls. Kevin Kelley of Anthropic, the draft’s author, wrote on the ai-control mailing list that it “complements our existing crawler controls rather than replacing them.”
Crawling is still governed by those older rules. The draft calls content-usage preferences and traffic signals “independent and orthogonal,” so whatever a GPTBot or ClaudeBot group says about fetching still applies. It also says “Private agreements MAY override preferences expressed via this mechanism,” which puts licensing deals above the signal. And the protocol underneath enforces nothing: RFC 9309 says its rules are not a form of access authorization, only rules crawlers are asked to honor. Anthropic itself has proposed robots.txt “do not crawl” signals to an Australian parliamentary inquiry, where SBS said in its own submission that ”Robots.txt mechanisms are routinely ignored or bypassed.”
What counts as training
The token follows the definition the IETF’s AI Preferences working group set out in its vocabulary draft, draft-ietf-aipref-vocab-08, dated September 13: “Using an asset to modify the learned parameters of a generative AI model.” Training models that perform “exclusively non-generative tasks” is excluded, even if those models can generate. The vocabulary also leaves out models used “for classification, ranking, or scoring, even where the model generates rationale for its output.”
When each backer starts
OpenAI aims to implement the signal “in the coming months” and promises an update with details when it does. Kelley wrote that Anthropic “has already started implementing support” and will roll it out over the coming months, adding that “how we interpret and apply these signals may change.” When we checked on November 2, Anthropic’s public help page still described only the ClaudeBot rule.
Microsoft will honor the new preferences from February 1, 2027, and Google’s crawler documentation says early 2027. So for now the one-line opt-out is a calendar, and its nearest firm date is three months away. SiliconANGLE reports that Meta supports the idea but has not committed to implementing it, and that OpenAI also plans controls similar to Microsoft’s grounding setting, described below.
Google-Extended and Bing’s grounding switch
Google adds a mapping. When a site opts out with the token and does not name Google-Extended explicitly, Google “plans to interpret that as if the publisher had disallowed Google-Extended.” Google’s documentation says Google-Extended governs both training and grounding, and its blog says sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” We read the two documents together to mean that a token-only opt-out would also take a site out of Gemini Apps grounding; Google does not say this in so many words. The mapping only applies where Google-Extended is not named, so an explicit group leaves that choice with the site.
Microsoft handles grounding separately, and only per page. Its Bing post says “These settings govern use of content, not crawl access” and “There is no site-wide robots.txt grounding preference.” The control is a meta tag such as <meta name="Bingbot" content="ai-grounding:disallow"> or an X-Robots-Tag header. Disallow cuts off grounding entirely. DisallowExtended limits it to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes the grounding services Microsoft provides to third parties, such as Web IQ and API grounding in Foundry. After March 1, 2027, Microsoft will no longer read NoArchive and NoCache as a preference for training or grounding, so pages that rely on those tags for that purpose need the new settings by then.
Our analysis
What we checked
On November 2 we read the Internet-Draft and its Datatracker entry, Kelley’s list post, Google’s and Microsoft’s posts and documentation, and Anthropic’s help page. We then ran the draft’s example file through three robots.txt parsers, once each.
SiliconANGLE reported that the four companies had sent the IETF a draft that blocks training scrapers and builds the standard into the Robots Exclusion Protocol format. In fact it is one Anthropic engineer’s individual draft. The working group has not adopted it, it leaves the format unchanged, and it records a preference about use while each company’s own rules still decide what gets crawled. In our single run, Python’s standard library also read the draft’s own example the opposite way from the other two parsers.
- The draft,
draft-kelley-aipref-training-preference-signal-00, is dated October 30, aims for Standards Track and expires May 3, 2027. Datatracker lists it as individual and says it “is not endorsed by the IETF and has no formal standing in the IETF standards process.” Staff from Google, Microsoft, OpenAI and Meta appear only in its acknowledgments. - It “introduces no new directive names and no new grammar,” and says: “This policy token is not a directive for crawling behavior.”
| Parser | Draft example, token on /press-kit/a |
|---|---|
| Python urllib.robotparser 3.10.12 | disallowed |
| protego 0.7.0 | allowed |
| Node robots-parser 3.0.1 | allowed |
All three parsers ignored the token group for other agents, as the draft intends. But Python’s standard library applies rules in file order, while RFC 9309 says “The most specific match found MUST be used,” and that is what the other two parsers do. When we moved Allow: /press-kit/ above Disallow: /, robotparser returned allowed too, matching protego.
Up next
Claude API enforces thinking blocks and moves to Rust
Anthropic’s API team didn’t have a single Rust developer when it ported the Messages API off a Python monolith. Claude wrote the code, and the guests put CPU savings at 40-50%. What share of your team’s code does Claude write today?
Related stories
- Anthropic asks Australia to allow AI training with opt-out
- OpenAI will honor one robots.txt signal for AI training
- Shared Claude chats were blocked with robots.txt but never tagged noindex
- Sweden’s Government Offices block Claude and ChatGPT
- FTC’s demands to Anthropic run over 50 pages each
- FTC plans to make Anthropic and OpenAI executives testify
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

