Google joins shared robots.txt AI opt-out, aims for 2027

Google said on October 30 that it plans to support a shared robots.txt signal for opting out of AI training. It will read that opt-out as if the site had disallowed Google-Extended, the AI training control it has offered since 2023. If you want your site out of training across vendors, this shared rule is the single step you will eventually use. Today, Engadget reports, it reaches only Anthropic, so a per-company rule such as a Google-Extended disallow is still what covers Google until its support arrives in 2027. Google says those rules keep working alongside the shared signal.

Engadget reports that four companies have committed to honoring the directive: Google, Anthropic, OpenAI and Microsoft. It comes from a proposal at the IETF written by Kevin Kelley, Anthropic’s technical lead for standards. The start dates differ. Anthropic already supports it, and Microsoft starts on February 1, 2027. OpenAI has promised the signal “in the coming months”. Engadget puts Google in early 2027, while Google’s own post says only that it aims to support the control along with other AI providers in 2027 and will give updates on its progress.

What the rule looks like

Right now a site that opts out has to identify each provider’s crawler and write a separate rule for it. In Engadget’s description, the shared version is a new user agent reserved for AI training: User-Agent: __usage_ai-training, followed by Disallow: / to block training crawlers site-wide and, if you want, a line like Allow: /press-kit/ to leave one section open to them. Google calls the same thing a policy token. It is built on the AI training definition from the IETF’s AIPREF working group, while the group keeps discussing the rest of its vocabulary.

The block does not stop crawling in general, and it does not take pages out of Google Search. Google applies it to Google-Extended only, and only when the file doesn’t name Google-Extended explicitly. A line written for Google-Extended therefore overrides the shared opt-out for Google, in either direction. That override matters in one place Google points to: sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. Grounding here means Gemini pulling in live web pages to back up its answers. So if you opt out of training everywhere but want to stay eligible there, you add an explicit Google-Extended allow next to the shared block.

Limits of a robots.txt line

Google describes the mechanism as a draft proposal that it will keep refining through the IETF working group. The post also says it is consistent with the EU Code of Practice for General Purpose AI model providers under the EU AI Act, which treats robots.txt as a primary way for publishers to express machine-readable preferences.

A preference is all it is. The AIPREF working group’s charter leaves technical enforcement of preferences out of scope, along with authenticating crawlers and auditing AI training. Whether a crawler obeys the line comes down to each provider’s commitment.

And the line only applies to future crawls. A page a company has already crawled stays in its hands, and the new directive cannot pull that content back.