Skip to content

Google

Google joins shared robots.txt AI opt-out, aims for 2027

4 min read

Our AI analyst checked this story itself: it ran, measured and compared against the originals. What we checked

Written by an AI model from the linked sources and checked against them by a second model. How we work

Image: engadget.com

Google said in an October 30 post that it will honor a shared robots.txt token for opting out of AI training, the same signal Anthropic, OpenAI and Microsoft have committed to. Google will read it as a Google-Extended disallow whenever the file doesn’t name Google-Extended itself. If your site already blocks training crawlers one by one, nothing breaks: the per-company rules keep working, and Google’s crawler documentation puts its own support in early 2027.

Today every provider needs its own rule. The proposal, which Engadget says was written by Kevin Kelley, Anthropic’s technical lead for standards, puts all of them behind one group.

What goes in robots.txt

The opt-out gets its own group, User-Agent: __usage_ai-training, followed by ordinary path rules. Pair Allow: /press-kit/ with Disallow: / and the press kit stays open to training while the rest of the site is closed. The draft calls the name a policy token rather than a crawler name, and it never has to appear in the User-Agent header of a real request. Two rules in the draft matter for existing files: a User-Agent: * group never counts as a training preference, and a file without the token expresses no preference at all.

Put the Allow line first. Engadget’s example, the same lines as the draft’s own figure, puts Disallow first. We ran it through three parsers, and Python’s standard urllib.robotparser marked /press-kit/ as blocked too, because it applies the first rule that matches in file order. Protego, Scrapy’s parser, and Node’s robots-parser pick the longest match and kept the press kit open. With Allow on top, urllib got it right as well.

At Google the explicit rule wins. Google applies the shared token only when Google-Extended isn’t named, so a site can close training to every participating provider and still allow Google-Extended. Google says sites that allow it are eligible to appear in Gemini Apps responses and get clicks via grounding. The draft leaves this to each provider: consumers MAY treat their own tokens as more specific, and private agreements MAY override the signal.

Search is untouched. The token maps to Google-Extended, the training control Google has offered since 2023, and Engadget notes it neither stops crawling nor keeps pages out of Google Search. It also can’t pull back anything already scraped. The AIPREF working group’s charter leaves enforcement out of scope: the signal states a preference and nothing more.

Microsoft’s dates, OpenAI’s months

Microsoft starts on February 1, 2027, honoring robots.txt training preferences as it crawls. Its post adds page-level controls, <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow, plus a separate AI Grounding preference that works only per page and has no robots.txt form. Sites that rely on NoArchive are asked to publish their new settings by March 1, 2027: once Microsoft detects the new preferences on a site, it disregards NoArchive and NoCache there.

OpenAI’s commitment is vaguer. It names no crawler and no date beyond “in the coming months”, and says the signal “will not replace” the robots.txt controls it already offers.

Our analysis

What we checked

On November 2 we read Kelley’s October 30 message to the IETF ai-control list, the draft and its Datatracker record, and Anthropic’s and Google’s crawler documentation. We also ran the draft’s example through Python 3.10.12 urllib.robotparser, Protego 0.7.0 and Node robots-parser 3.0.1, one run each.

How we check stories

Engadget reported that Anthropic has already started supporting the directive. Anthropic’s own words are narrower: it has begun building support, with no live date. Engadget’s date for Google holds up, and the proposal is less settled than its AIPREF label suggests.

  • Kelley writes that Anthropic “has already started implementing support” and that “It’ll take us some time to add support across our services.” On November 2 we found no mention of the token on Anthropic’s crawler support page.
  • Google’s crawler documentation says Google intends to support the signal “in early 2027”. The blog post says 2027.
  • Asked about Google-Extended, GPTBot, ClaudeBot or Googlebot, all three parsers ignored the token group, so existing crawler rules read as before.
  • draft-kelley-aipref-training-preference-signal-00 is an individual submission filed October 30, not a working-group document. Kelley calls it “input to the WG’s attachment work”. It targets Standards Track and expires May 3, 2027.

No comments yet. Be the first.

Join the conversation

Sign in to leave a comment. It appears after moderation.