OpenAI, Google and Microsoft back one robots.txt line to opt out of AI training

On October 30, OpenAI, Google and Microsoft each said they will honor one shared robots.txt signal that tells AI companies whether a website’s content may be used to train generative models. The signal is a policy token, __usage_ai-training, defined in an IETF Internet-Draft dated the same day and written by Kevin Kelley of Anthropic.

Until now, a site that wanted to stay out of training data had to learn each company’s crawler names and write a separate rule for each one. Google’s control, for example, is Google-Extended, which it has offered since 2023. The new token lets a site state one preference that every participating provider reads.

What the line looks like

The draft adds no new robots.txt grammar. A publisher writes an ordinary group and puts the token where a crawler name would normally go. The draft’s own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That expresses a preference against training for the whole site, except the press kit folder. Two details are easy to miss. A wildcard group (User-Agent: *) does not count as matching the token, so existing catch-all rules say nothing about training. And the token governs use, not access:

This policy token is not a directive for crawling behavior.

That line is from section 4.4 of the draft. A crawler can still fetch the page; the provider is being asked not to train on it. If no group names the token, the draft says no preference has been expressed, and each company’s own crawler rules apply as before.

The definition of training comes from the IETF AI Preferences (AIPREF) working group’s vocabulary: using content to change the learned parameters of a generative model. Training models used only for classification, ranking or scoring falls outside it.

Same signal, different dates

The three announcements agree on the token but not on the schedule.

OpenAI says it aims to implement the signal “in the coming months” and will publish details when it does. Its existing robots.txt controls stay in place, and it asks other AI providers to honor the signal too.

Google’s John Mueller wrote that Google aims to support the control in 2027. Google adds one rule of its own: if a site opts out with the shared token and does not mention Google-Extended, Google will treat it as if Google-Extended were disallowed. Mueller also points to the EU’s Code of Practice for general-purpose AI model providers, which recognizes robots.txt as a main way for publishers to state machine-readable preferences. That gives the companies a regulatory reason to agree on one format.

Microsoft published the most detail. Bing will start honoring the new preferences as it crawls from February 1, 2027. It also adds page-level tags, ai-training:disallow and ai-grounding:disallow, set through a Bingbot meta tag or an X-Robots-Tag header. The older NoArchive and NoCache directives stop counting as training or grounding preferences after March 1, 2027. Bing product manager Krishna Madhavan put the goal this way:

Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.

Anthropic wrote the draft, but its newsroom carried no separate rollout date as of November 3.

What the token does not cover

The shared signal is about training only. Whether a chatbot may read a page live and summarize it in an answer, usually called grounding, is a separate question, and each company handles it differently. Microsoft built its own page-level grounding control and says there is no site-wide robots.txt setting for it. Google ties grounding in Gemini apps to Google-Extended: a site that explicitly allows it can still appear in answers and receive clicks. OpenAI’s post does not address grounding.

The draft also looks forward, not back. It says the preference that counts is the one in the robots.txt file “in effect when the asset was retrieved.” It does not say what happens to pages collected before a site adds the line.

And the document is not a standard. It is an individual Internet-Draft with no formal standing at the IETF. Nate Hake, who says he has spent the past year in the working group, noted in replies quoted by Search Engine Roundtable’s Barry Schwartz that “some significant areas of dispute remain.” His main objection is that the current text “basically exempts the preferences from applying to anything a user does,” which he argues hurts creators facing AI copycats.

What site owners can do now

Adding the token costs nothing. It expresses a preference about training, not a crawling rule, and Microsoft states that sites remain eligible for Bing results whatever they choose. But no provider is reading the token yet, so anyone who wants to stay out of training today should keep their per-company rules alongside it.

Sites that rely on NoArchive or NoCache to keep content out of Bing’s AI features have a firmer deadline. They need to set the new preferences, and for grounding record a temporary choice in Bing Webmaster Tools, before March 1, 2027.

The bigger shift is who wrote the rule. The draft’s acknowledgments list people from AI labs, Cloudflare and publishers including the Financial Times and News Corp, with affiliations given for identification only. If the working group settles the disputes Hake describes, small sites would get a single line instead of a growing list of bot names. Until then, the token is a promise from three companies, on three different timetables.