Anthropic joins a shared robots.txt opt-out for AI training

OpenAI said on October 30 that it will honor a shared robots.txt signal that lets a website opt out of generative AI training at several providers with one entry, and SiliconANGLE reports that Google, Microsoft and Anthropic back the same standard. The standard is still upcoming, so the per-crawler rules sites keep today are what counts until each provider ships support: OpenAI and Microsoft plan to add it in the coming months and Google next year, per the same report, and OpenAI says its existing crawler controls will keep applying afterwards.

The work it removes is bookkeeping. An operator who wants out of training now has to identify each provider’s crawler and maintain a separate rule for it, and SiliconANGLE notes that a provider running several scrapers needs an entry for every one of them. The shared signal collapses that into a single robots.txt entry covering all the scrapers that gather training data for generative models. OpenAI calls it a voluntary effort and says it will publish details of its implementation when it ships.

Training is the only thing the signal covers. It won’t apply to models that “perform exclusively non-generative tasks,” SiliconANGLE says, and pages an AI fetches to answer a user’s question sit under a different control. Microsoft said in a blog post that it will add a setting called AI Grounding for exactly that case: operators can block such requests from its services, and the block extends to third-party AI tools that use Microsoft technology to process prompts. OpenAI plans similar controls.

The signal draws on the work of the IETF’s AI Preferences working group, the standards body’s group on how sites state preferences about AI use of their content. SiliconANGLE writes that the backers have already submitted a draft to the IETF, which maintains the Robots Exclusion Protocol, the format robots.txt follows. The current version of that protocol, RFC 9309 from September 2022, states that its rules are not a form of access authorization but requests crawlers are asked to honor, and nothing in the system enforces them. The new entry will bind only the crawlers whose operators choose to read it.

That limit has already surfaced in policy fights. In a submission to an Australian parliamentary inquiry, Anthropic proposed “do not crawl” signals in robots.txt, while SBS wrote in its own submission that “Robots.txt mechanisms are routinely ignored or bypassed.”

OpenAI’s post ends by urging other AI providers to honor the signal. Meta, SiliconANGLE reports, supports the idea behind it but hasn’t committed to implementing it.