Google, Microsoft, Anthropic and OpenAI said on 30 October that they will back a shared way for websites to refuse AI training: one robots.txt entry that would apply across participating AI developers. A draft of the proposal has already gone to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol behind the file.
“Neither choice guarantees inclusion, traffic, or payment.”
Microsoft, Bing blog post, on allowing or blocking AI training and grounding · blogs.bing.com ↗
At present a site must list every AI provider separately, and each extra crawler a provider runs needs its own line. The new signal would cover every scraper gathering training data for generative models, while models doing only non-generative work fall outside it. Where a site uses the token and has not named Google-Extended, Google plans to read it as a Google-Extended disallow, and its own controls keep working alongside.
Microsoft's graphic for its post on separate training and grounding preferences
Image: Microsoft Bing Blog
Microsoft is adding a separate grounding preference, for pages an AI reads to answer a question rather than to train. Set page by page, Disallow stops that use, while DisallowExtended keeps it to Microsoft's own products. Both default to allow. Microsoft warns that blocking grounding lowers a brand's visibility in AI experiences.
About robots.txt
robots.txt is a plain text file on a website that tells crawlers which content they may access. Site owners can use it to block scraping across the whole site or only for chosen pages. The Robots Exclusion Protocol behind it was first defined by Martijn Koster in 1994.
The Internet Engineering Task Force published the protocol as RFC 9309, a Proposed Standard, in September 2022, with Koster and Google's Gary Illyes among its authors. The file's role in AI began to grow in 2023: Google introduced Google-Extended, a robots.txt control over whether a site's content is used for AI training, and Microsoft brought in its NoArchive and NoCache publisher controls for AI training and grounding.
Microsoft will honor the new preferences from 1 February 2027, and its NoArchive and NoCache tags stop counting after 1 March 2027. Google aims for 2027, OpenAI says only the coming months, and Meta backs the idea without committing to use it. The IETF's AI Preferences working group is still working toward a final standard.