OpenAI will honor one robots.txt signal for AI training

OpenAI said on October 30 that it will honor a shared signal in robots.txt. The signal tells every participating AI provider at once whether a site’s content may be used for foundational generative AI training, so site owners won’t need a separate rule for each provider’s crawler. The signal isn’t live yet. OpenAI plans to implement it “in the coming months,” so for now, opting out of OpenAI’s training still means a rule for GPTBot.

GPTBot is the crawler that collects training data. OpenAI runs two other crawlers alongside it: OAI-SearchBot, which surfaces sites in ChatGPT search results, and ChatGPT-User, which fires when a person asks ChatGPT to visit a page. Disallowing GPTBot tells OpenAI not to use the content to train its generative foundation models, and it does nothing to the other two. Other providers split their crawlers the same way. Anthropic lists ClaudeBot for training plus Claude-SearchBot and Claude-User, and blocking ClaudeBot alone does not block Claude-SearchBot. Google-Extended is a control token, not a crawler. It decides whether crawled content can train Gemini Apps and Grounding with Google Search on Vertex AI, and it has no effect on Google Search, AI Overviews or AI Mode. Keeping all of these rules in sync is the work the shared signal is meant to remove.

OpenAI says the signal will not replace its current robots.txt controls, which “will continue to apply and be respected.” A GPTBot rule written today keeps working after the signal arrives. The signal covers training only, the same job the GPTBot rule does, and it is voluntary. It reaches only the providers that choose to honor it. It comes out of the ongoing work of the IETF’s AI Preferences working group, and other participants in the group announced it together with OpenAI. OpenAI also urged other AI providers to honor it and said it will post implementation details once the signal ships.

A training preference also won’t reach fetches that a user triggers. ChatGPT-User fires when someone asks ChatGPT to open a page, so robots.txt rules may not apply to it at all.