OpenAI will honor one robots.txt signal for AI training

OpenAI said on October 30 it will honor a shared robots.txt signal for opting out of foundational generative AI training. Google, Microsoft and Anthropic also plan to read the signal, so a site will eventually set one training preference for all four instead of keeping a rule for each crawler. For now nothing changes: OpenAI promises support “in the coming months,” and its existing crawler controls keep applying alongside the voluntary signal, OpenAI’s announcement says.
The signal is an ordinary robots.txt group with a reserved name where a crawler’s name would go: User-Agent: __usage_ai-training, then Disallow: / to object to training, or Allow lines for the paths you are fine with. The spec’s own example disallows the whole site and allows /press-kit/. It adds no new directives or grammar. A User-Agent: * group must not be read as matching it, so a robots.txt without this group expresses no preference at all.
Four providers, four clocks
Kevin Kelley of Anthropic, who wrote the spec, says the company “has already started implementing support,” and that it “complements our existing crawler controls rather than replacing them.” Microsoft will honor the preference as it crawls starting February 1, 2027. Google aims to support it in 2027, and its crawler documentation narrows that to early 2027. OpenAI gives only “the coming months” and says it will publish implementation details when it ships.
Until then, only the per-crawler rules do anything. GPTBot, ClaudeBot and Google-Extended keep working as they do now, and OpenAI says its existing controls will still be respected once the shared signal arrives.
The Google-Extended catch
The token is defined as a training preference, but Google’s mapping reaches further. When a site opts out through __usage_ai-training and does not mention Google-Extended, Google plans to treat that as a disallow of Google-Extended. That token also governs grounding (feeding Search-index content to the model at prompt time) in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says sites that specifically allow Google-Extended stay eligible to appear in Gemini Apps answers and get clicks from them. Under Google’s stated plan, a site that opts out through the shared token alone, with Google-Extended left unset, would lose Gemini Apps grounding unless it adds an explicit Google-Extended allow. This comes from Google’s documentation, not from a test: the token is not live yet.
Microsoft is adding a second preference of its own, a page-level “AI Grounding” setting with the values Allow, DisallowExtended and Disallow. DisallowExtended limits grounding to Microsoft-operated experiences such as Copilot and Microsoft 365. Sites that rely on NoArchive or NoCache as an AI signal have until March 1, 2027 to move; after that date Microsoft will no longer read them as a training or grounding preference.
Kelley’s draft also sets precedence rules. A provider may document that it treats its own token as more specific than the shared one, and, in the draft’s words, “Private agreements MAY override preferences expressed via this mechanism.”
Our analysis
What we checked
On October 31 we fetched /robots.txt from the top 1,000 domains of the Tranco ranking (list 746VX), one request per apex domain, and parsed the 499 readable files. Domains that failed to resolve or returned 403 are not counted.
OpenAI says the signal draws on the IETF AI Preferences working group’s work. In fact the mechanism is an individual Internet-Draft Kelley filed the same day, which has no formal standing in the IETF, and it takes only the group’s definition of AI training. The per-provider upkeep OpenAI describes shows up in our sample: 149 of the 499 sites disallow at least one of nine AI-training tokens, and 116 do so for three or more. None uses the new group yet, while 15 already carry Cloudflare’s competing Content-Signal line and one, launchpad.net, uses the working group’s Content-Usage form.
| Token | Named in a group | Fully blocked (Disallow: /, no Allow) |
|---|---|---|
| GPTBot | 124 | 60 |
| ClaudeBot | 119 | 60 |
| Google-Extended | 106 | 49 |
| CCBot | 109 | 66 |
| Bytespider | 102 | 71 |
| __usage_ai-training | 0 | 0 |
Of those 499 files, 39 fully block GPTBot, ClaudeBot and Google-Extended at once: three separate rules covering three of the providers that have now signed up to read one shared group.
Up next
Claude Ultra drops the 5-hour limit, keeps the $200 price
“No 5-hour limit” now sits on Claude’s $200 Ultra plan, the renamed Max 20x, for verified accounts. The weekly cap stays at 10x Pro usage, though. Has the 5-hour limit ever cut you off mid-task in Claude?
Related stories
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

