Select Page

Robots.txt AI Training Opt-Out: What the New Shared Signal Means

by | Nov 2, 2026 | AI

Conceptual illustration of website content passing through a permission gateway towards an AI network

Website owners may soon have a simpler way to tell several AI companies that their content should not train generative models. On 30 October 2026, Google, Microsoft and OpenAI announced support plans for a shared preference in robots.txt.

The proposed robots.txt AI training opt-out could reduce the burden of maintaining separate rules for every provider. However, it is not a universal blocking tool, and announced support does not mean every company already honours it. Here is what publishers, developers and Australian businesses should understand.

Background: crawling, training and grounding are different

Robots.txt traditionally communicates crawler-access preferences. AI has complicated that picture because collecting a page, training a model on it and consulting it while answering a question are different activities.

Training changes a generative model’s learned parameters. Grounding supplies outside content when an answer is produced. Traditional search indexing is another use again. A publisher might welcome search referrals while declining training, or permit cited AI answers without allowing its archive into future models.

Existing provider-specific controls offer different combinations of these choices. The shared proposal aims to simplify the training preference, not erase those differences.

What the new shared signal would change

An individual Internet-Draft submitted by Anthropic’s Kevin Kelley describes a policy token named __usage_ai-training. Website operators would address an ordinary robots.txt group to that token, using familiar Allow and Disallow rules.

A proposed whole-site preference

User-Agent: __usage_ai-training
Disallow: /

Under the draft, this expresses a preference against AI training across the site. More specific paths can have different preferences. A wildcard group addressed to User-Agent: * does not count as the shared signal, and a file without the named token expresses no preference through this mechanism.

Crucially, the token concerns content use rather than crawl access. It should not be presented as a firewall. The IETF Datatracker also labels the document an individual draft without IETF endorsement or formal standing in its standards process: this is work in progress, not a finished standard.

Rollout dates and provider differences

Google: planned for early 2027

Google’s announcement targets 2027, while its crawler documentation specifies early 2027. Where a site publishes the shared training opt-out without an explicit Google-Extended rule, Google plans to treat it as a Google-Extended disallow.

That distinction matters. Google-Extended governs both future Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says the control does not affect inclusion or ranking in Google Search. Publishers should therefore assess Gemini grounding separately from traditional search visibility before selecting an exception.

Microsoft: new preferences from 1 February 2027

Microsoft plans to honour the new preferences as it crawls from 1 February 2027. It separates AI Training from AI Grounding and adds page-level training settings that can override the robots.txt training preference.

Its grounding choices include allowing use, disallowing it, or restricting it to Microsoft-operated experiences through DisallowExtended. Microsoft says these preferences do not change eligibility for traditional Bing results, but restrictive grounding choices reduce visibility in AI-generated experiences. Allowing either use guarantees neither traffic nor payment.

Sites using NoArchive or NoCache should pay particular attention: after 1 March 2027, Microsoft will stop reading those tags as training or grounding preferences. Eligible existing users seeking continued grounding restrictions must publish page-level preferences and record a temporary transition preference in Bing Webmaster Tools before that deadline. The temporary option opens on 15 December 2026 for sites using the old tags on that date.

OpenAI: implementation in the coming months

OpenAI says it aims to implement the shared signal in the coming months and will publish implementation details. Its existing robots.txt controls will remain in effect.

OpenAI’s crawler documentation already distinguishes GPTBot, associated with potential training use, from OAI-SearchBot, which supports ChatGPT search. Those settings are independent. The new announcement should not be mistaken for a reason to remove working provider-specific rules today.

Practical steps for publishers and businesses

The immediate benefit is a clearer planning framework, not a requirement to rush out a robots.txt change. For an Australian small business, agency or publisher, a sensible preparation checklist is:

  • Inventory existing rules. Record crawler-specific groups, Google-Extended settings, and any NoArchive or NoCache tags.
  • Decide the policy first. Document whether training, cited AI answers and ordinary search discovery should be allowed for each important content section.
  • Review provider exceptions. Check whether preserving grounding eligibility also permits training under that provider’s controls.
  • Assign an owner and review dates. Have a developer validate proposed rules against current documentation and track Microsoft’s transition if applicable.

Agencies should keep these decisions separate for each client rather than copying one configuration everywhere. A news archive, an online shop and a public documentation site can have different priorities.

Risks and limitations

The shared signal relies on participating providers interpreting and respecting it. It does not technically prevent a non-compliant scraper from requesting public pages. Sensitive material still needs appropriate access controls.

Nor should a new preference be assumed to remove content from existing models. The draft ties preferences to the robots.txt file in effect when content was retrieved. Changing today’s file is not a documented retroactive deletion mechanism.

Implementation also takes time. Microsoft explicitly notes that publishing a setting does not activate it until crawling discovers it. Businesses should avoid promising immediate protection or unchanged AI referrals after an edit.

What to watch next

Watch for OpenAI’s implementation notice, Google’s early-2027 documentation updates, and revisions to the Internet-Draft. Microsoft users should also review its detailed migration guidance before the February rollout and March deadline. The important test is documented provider behaviour, not simply whether a token appears in a file.

Conclusion

A shared robots.txt AI training opt-out is a useful simplification of a fragmented system. It is not yet a single, fully deployed switch with identical effects everywhere. Keep existing controls, distinguish training from grounding, and plan changes around verified implementation details.

Sources