Google joins shared robots.txt AI opt-out, aims for 2027

Google said in an October 30 post that it will honor a shared robots.txt token for opting out of AI training, the same signal Anthropic, OpenAI and Microsoft have committed to. Google will read it as a Google-Extended disallow whenever the file doesn’t name Google-Extended itself. If your site already blocks training crawlers one by one, nothing breaks: the per-company rules keep working, and Google’s crawler documentation puts its own support in early 2027.
Today every provider needs its own rule. The proposal, which Engadget says was written by Kevin Kelley, Anthropic’s technical lead for standards, puts all of them behind one group.
What goes in robots.txt
The opt-out gets its own group, User-Agent: __usage_ai-training, followed by ordinary path rules. Pair Allow: /press-kit/ with Disallow: / and the press kit stays open to training while the rest of the site is closed. The draft calls the name a policy token rather than a crawler name, and it never has to appear in the User-Agent header of a real request. Two rules in the draft matter for existing files: a User-Agent: * group never counts as a training preference, and a file without the token expresses no preference at all.
Put the Allow line first. Engadget’s example, the same lines as the draft’s own figure, puts Disallow first. We ran it through three parsers, and Python’s standard urllib.robotparser marked /press-kit/ as blocked too, because it applies the first rule that matches in file order. Protego, Scrapy’s parser, and Node’s robots-parser pick the longest match and kept the press kit open. With Allow on top, urllib got it right as well.
At Google the explicit rule wins. Google applies the shared token only when Google-Extended isn’t named, so a site can close training to every participating provider and still allow Google-Extended. Google says sites that allow it are eligible to appear in Gemini Apps responses and get clicks via grounding. The draft leaves this to each provider: consumers MAY treat their own tokens as more specific, and private agreements MAY override the signal.
Search is untouched. The token maps to Google-Extended, the training control Google has offered since 2023, and Engadget notes it neither stops crawling nor keeps pages out of Google Search. It also can’t pull back anything already scraped. The AIPREF working group’s charter leaves enforcement out of scope: the signal states a preference and nothing more.
Microsoft’s dates, OpenAI’s months
Microsoft starts on February 1, 2027, honoring robots.txt training preferences as it crawls. Its post adds page-level controls, <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow, plus a separate AI Grounding preference that works only per page and has no robots.txt form. Sites that rely on NoArchive are asked to publish their new settings by March 1, 2027: once Microsoft detects the new preferences on a site, it disregards NoArchive and NoCache there.
OpenAI’s commitment is vaguer. It names no crawler and no date beyond “in the coming months”, and says the signal “will not replace” the robots.txt controls it already offers.
Our analysis
What we checked
On November 2 we read Kelley’s October 30 message to the IETF ai-control list, the draft and its Datatracker record, and Anthropic’s and Google’s crawler documentation. We also ran the draft’s example through Python 3.10.12 urllib.robotparser, Protego 0.7.0 and Node robots-parser 3.0.1, one run each.
Engadget reported that Anthropic has already started supporting the directive. Anthropic’s own words are narrower: it has begun building support, with no live date. Engadget’s date for Google holds up, and the proposal is less settled than its AIPREF label suggests.
- Kelley writes that Anthropic “has already started implementing support” and that “It’ll take us some time to add support across our services.” On November 2 we found no mention of the token on Anthropic’s crawler support page.
- Google’s crawler documentation says Google intends to support the signal “in early 2027”. The blog post says 2027.
- Asked about Google-Extended, GPTBot, ClaudeBot or Googlebot, all three parsers ignored the token group, so existing crawler rules read as before.
draft-kelley-aipref-training-preference-signal-00is an individual submission filed October 30, not a working-group document. Kelley calls it “input to the WG’s attachment work”. It targets Standards Track and expires May 3, 2027.
Up next
Claude Ultra drops the 5-hour cap but keeps the weekly one
Claude’s $200 plan, renamed Ultra, just dropped the 5-hour cap for verified accounts, but the weekly allowance stays at 10 times Pro’s. Would you pay $200 to skip the 5-hour cap, or keep waiting out the reset?
Related stories
- OpenAI will honor one robots.txt signal for AI training
- Sweden’s Government Offices block ChatGPT and Claude
- Anthropic seeks en banc rehearing over Pentagon risk label
- Anthropic wrote its own AI safety bill for Congress
- FTC’s subpoenas to Anthropic and OpenAI run past 50 pages
- Anthropic whistleblower testimony airs on NDTV Profit
Comments
No comments yet. Be the first.
Join the conversation
Sign in to leave a comment. It appears after moderation.

