🔍 Read the full analysis: How To Set The Opt-Out Supported By OpenAI, Google And Microsoft on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal for publishers who do not want their content used to train generative AI models. The signal is voluntary, does not block crawling or AI answers, and will roll out on different schedules; Meta has voiced support without committing to honor it.
OpenAI, Google, Microsoft and Anthropic said they will support a shared robots.txt signal that lets website publishers request that their content not be used to train generative AI models. The proposal, published as an Internet-Draft on October 30, is intended to replace the need for separate training opt-outs for each participating company, but it is a voluntary request rather than a technical block.
The proposed rule uses the user-agent name __usage_ai-training, followed by Disallow: /. A publisher adding that group to its robots.txt file is expressing a preference against training use. The companies have not adopted one common start date: Anthropic says it has begun implementation, OpenAI aims to support the signal in the coming months, Google aims for 2027, and Microsoft says it will honor it from February 1, 2027, as it crawls sites.
Google’s Search Central blog says that if a publisher uses the new signal and has no separate Google-Extended rule, Google plans to treat the signal as a disallow for Google-Extended. Google’s crawler documentation describes its target as early 2027. OpenAI calls its participation a voluntary effort covering foundational generative AI training and says its existing crawler controls will remain in effect.
Microsoft’s announcement also covers a separate page-level preference for AI Grounding: use of web content to inform answers in Bing, Copilot and Microsoft 365. Both Microsoft preferences take effect on February 1, 2027. Microsoft says NoArchive and NoCache will stop counting as training or grounding preferences after March 1, 2027. Anthropic says implementation is underway but will take time across its services. Meta has expressed support for a shared signal, but has not said whether it will honor it.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
What Publishers Can Control
The shared rule could make it simpler for publishers to communicate a training preference across several major AI companies. Instead of maintaining a separate training instruction for each participating crawler, a site could add one common rule while companies build support for it. That may reduce configuration work, though publishers will need to check each provider’s implementation and keep existing controls in place during the rollout.
The distinction between training and AI answers matters. The proposed signal addresses training, not the use of live website content to answer a query. Microsoft’s separate AI Grounding preference makes that difference explicit, but the other companies’ announcements do not establish that the shared training token controls answer-generation uses. The rule also does not promise payment, traffic or inclusion in a product; Microsoft says neither preference guarantees those outcomes.
For readers and site operators, the key practical point is that robots.txt expresses a preference; it does not impose a universal enforcement mechanism. The companies’ commitments and crawler behavior will determine whether the request is followed. The signal also does not remove material from models that have already been trained.
Products Worth Considering
Bose QuietComfort Wireless Noise Cancelling Headphones, Sandstone
LG UltraGear 32GX850A-B 32-Inch 4K UHD Glossy OLED Gaming Monitor
DJI Osmo Mobile 8 3-Axis Phone Gimbal Stabilizer
As an affiliate, we earn on qualifying purchases.
The proposal is titled “Machine-Readable Training Preference Signal” and is identified as draft-kelley-aipref-training-preference-signal-00. It was written by Anthropic standards lead Kevin Kelley and published within work associated with the Internet Engineering Task Force’s AI Preferences working group. An Internet-Draft is a proposal for discussion, not an adopted internet standard.
The working group’s status remains unsettled. At a meeting on October 21, one of its chairs said, “We haven’t declared consensus on anything yet.” The draft itself has no formal standing in the IETF standards process. Support from companies therefore does not mean that the rule has become an official standard or that every AI provider will use it.
Existing company-specific controls remain relevant. The announcements say that current crawler rules, including GPTBot, Google-Extended and ClaudeBot, continue to apply. Microsoft also says its preferences do not change eligibility for Bing search results, while Google says Google-Extended does not affect Google Search. Publishers should distinguish among training, AI answer uses and ordinary search visibility rather than assuming one instruction covers all three.
““has already started implementing support for this draft””
— Anthropic standards lead Kevin Kelley
Adoption and Scope Remain Open
Participation is incomplete and uneven. Anthropic, OpenAI, Google and Microsoft have announced support, with different implementation schedules. Meta has voiced support but made no commitment to follow the signal. The source material reports no statement from Apple, Amazon, xAI or Perplexity; that does not establish whether they will adopt it later.
It is also not yet clear how consistently the rule will be interpreted across services or how quickly each company’s crawlers and products will reflect it. The proposal is voluntary and not an adopted IETF standard. Publishers should not assume that adding the rule immediately prevents use by every company, or that it affects existing training data, live-web answers or search indexing.
Watch the Rollout Dates
Publishers considering the opt-out can add the shared rule while retaining existing company-specific training instructions until each provider has implemented the common signal. They should separately review controls for AI answers and search, since those uses are not all covered by the training preference.
The next milestones are the companies’ implementation rollouts: OpenAI says “in the coming months,” Google targets 2027 and specifies early 2027 in its crawler documentation, and Microsoft’s preferences begin February 1, 2027. Sites using Microsoft’s NoArchive or NoCache tags for training or grounding preferences should account for the announced March 1, 2027 cutoff. Further statements from Meta and other providers, along with any changes in the IETF working group’s status, will clarify how broadly the signal is adopted.
Key Questions
What should a publisher add to robots.txt?
The proposed group is User-Agent: __usage_ai-training with Disallow: /. It expresses a preference against using the site’s content for generative AI training by companies that support the signal.
Does the rule block AI companies from crawling a website?
No. The source describes it as a training-use preference, not a block on crawling. It also does not remove content from models that have already been trained.
Will it stop websites from appearing in AI answers?
Not by itself, according to the announcements described here. The shared signal is for training. Microsoft has announced a separate AI Grounding preference for content used in answers from Bing, Copilot and Microsoft 365.
When will the companies honor the signal?
Dates vary. Anthropic says implementation has started; OpenAI aims to support it in the coming months; Google targets 2027, with documentation specifying early 2027; Microsoft starts February 1, 2027.
Has the signal become an official internet standard?
No. It is an Internet-Draft and has no formal standing in the IETF standards process. A working-group chair said on October 21 that the group had not declared consensus.
Source: ThorstenMeyerAI.com
