# OpenAI and Google Plan One AI Training Opt-Out While Search Stays Open

> OpenAI and Google backed a shared robots.txt signal for AI training on October 30. Publishers still face separate decisions about search visibility, AI answers, existing bot rules and paid reuse.

- Published: 2026-10-31
- Author: Gene Dai
- Canonical: [https://digidai.github.io/2026/10/31/shared-ai-training-opt-out-publishers-search/](https://digidai.github.io/2026/10/31/shared-ai-training-opt-out-publishers-search/)
- Topics: Artificial Intelligence, Publishing, Search, AI Training, Deep Investigation

---

![A publisher's paper archive branches toward a search window and a closed training folder beneath the words One Signal, Two Uses and a small Gene Dai signature.](/images/articles/shared-ai-training-opt-out-publishers-search/cover-v1.jpg)

_AI-generated editorial illustration._

On October 30, a publisher checking its `robots.txt` file saw two promises with different clocks.
[OpenAI said](https://openai.com/index/helping-websites-express-ai-training-preferences/) it would honor a shared signal
for generative AI training in the coming months.
[Google said](https://developers.google.com/search/blog/2026/10/ai-training-control) it planned support in 2027. The
signal begins with `User-Agent: __usage_ai-training` and uses ordinary path rules. One file could eventually state a
training preference to several providers.

The announcement did not switch on an industry standard overnight. Its underlying
[October 30 Internet-Draft](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) is an
individual proposal by Anthropic's Kevin Kelley, published through the IETF process. The IETF's own record says it has
no formal standing as an approved standard. OpenAI gave no exact implementation date. Google described a plan for 2027.
A publisher editing the new line today cannot assume every crawler has changed its behavior.

The publisher has to make today's choice with today's tools. It wants people to find an article through search and
perhaps through an AI answer. It may also want to sell a training license. A `Disallow` rule aimed at every crawler can
sacrifice discovery; an open door to discovery does not settle training rights or payment. The new signal separates one
use of the work from the act of fetching it. It leaves the price blank.

A site editor can ask the search lead for a list of pages that must remain discoverable. The licensing lead may return a
different list of pages that can be sold for training. The two lists will overlap, and neither person can resolve the
overlap by changing one bot rule. The draft supplies a way to state a training preference. Search access, answer
appearance and payment still need their own controls and records.

## October 30 produced a promise, not a switch

OpenAI's announcement is short. It says a common signal would reduce the work of naming and maintaining separate crawler
rules for each participating provider. Its existing `robots.txt` controls continue. The company intends to implement the
shared signal in the coming months and provide more detail then. It does not publish a rollout percentage, a testing
receipt for any publisher or a promise that a training exclusion will generate traffic.

Google's announcement sets a later date and describes a particular interaction with its own control. If a site opts out
through the new policy token and does **not** explicitly specify `Google-Extended`, Google plans to interpret the signal
as if `Google-Extended` were disallowed. If the site does specify `Google-Extended`, that vendor-specific instruction
can express a different choice. Google says its support is planned for 2027. That is a stated future interpretation, not
evidence that its crawler applies the new token today.

John Mueller, Google's Search Advocate, signed the Google post. The detail about `Google-Extended` is the part a
publisher cannot skip. A generic opt-out may preserve ordinary Google Search while changing eligibility for Gemini
grounding. A publisher that relies on referrals from both products needs to decide whether those channels deserve the
same policy. Google's announcement describes that choice; it does not forecast how many visits either route would send.
A search team would have to measure that on the site's own pages after implementation.

Kevin Kelley works at Anthropic, and the draft's acknowledgments name participants from publishers, technology companies
and civil society. Those affiliations do not turn his individual draft into an Anthropic product announcement or IETF
consensus. An editor writing a policy this week faces a simpler fact: a draft can change, and neither provider has
switched on its announced support. Existing bot controls remain the operative published instructions until a provider
documents and demonstrates the new behavior.

A public `robots.txt` line can prove what the site declared on a date. It cannot prove what a model developer collected
months earlier, whether a mixed-purpose crawler retained data for a later use, or whether a dataset provider passed a
restriction along. A newsroom that adds the line should record the date and the served file. It should not report that
training stopped. That claim requires evidence the public file cannot provide.

## One token describes a use, not a visitor

The [draft text](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-00) uses the
familiar `robots.txt` structure. Its example places `User-Agent: __usage_ai-training` above `Disallow: /` and then
`Allow: /press-kit/`. Under that example, a service operator expresses a preference against generative AI training for
the site except the press-kit path. The `Allow` and `Disallow` syntax does not change. The new item is the purpose token
in the user-agent position.

That position is easy to misread. The policy token does not have to appear in the HTTP request's user-agent header. It
is a key a participating provider uses when interpreting the file. A site's access logs might therefore show Googlebot
or another real crawler, while the use preference sits in a separate policy group. Looking for `__usage_ai-training` as
though it were a visiting bot would miss the point.

The draft defines the covered use as using an asset to modify the learned parameters of a generative AI model. It
explicitly leaves exclusively non-generative classification, ranking and scoring outside that category. It does not
claim to decide whether a particular answer's retrieval, quotation, or display is training. A policy against one use
cannot be read as a universal veto over every kind of AI processing.

That narrow definition can be useful even to a publisher that does not want a total block. A job board might want its
public listings found by a search service while keeping proprietary descriptions out of model training. A software
company may want freely available documentation retrieved in answers but reserve a paid training corpus. These are
possible policy choices, not accounts of named companies changing their files. The policy has to follow the rights in
the actual work. A site that states a training preference for content it merely hosts under license may be declaring a
choice the underlying rights holder never made.

The text also says wildcard `User-Agent: *` groups must not be treated as matching the policy token. No group with the
new token means no preference expressed **through this mechanism**. That is not a statement that every use is
authorized. Other terms, laws and vendor-specific signals may still matter. The draft treats crawler access rules and
content-use preferences as separate. A provider may choose not to fetch a page it cannot train on, but the token itself
is not an instruction requiring it to stop crawling.

The draft leaves precedence partly in the consumer's hands. More specific expressions should generally outrank broad
ones, it says, and a provider may document that its own token is more specific than the shared signal. A private
agreement may also override a public preference. Those qualifications make a site-wide copy-and-paste recipe risky. An
operator with a `Google-Extended` rule or a direct data license must read the contract and the provider's interpretation
together. Fewer lines in the file do not mean fewer decisions.

For a site with multiple rights holders, path rules are useful only if the URL structure follows the rights map. A
publisher might control its original reporting but license syndicated photographs under different terms. If both assets
appear under a common path, a single broad rule might misstate the rights it can grant or reserve. The draft's path
example makes the need for an inventory visible. It does not do that inventory for the operator.

## Search access already has separate controls

OpenAI's [crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes `GPTBot`, associated with
training its generative models, from `OAI-SearchBot`, used to surface sites in ChatGPT search. It says their settings
are independent. A site can disallow GPTBot while allowing OAI-SearchBot. It also says it may reuse the result of one
crawl for both uses when both are allowed, to avoid duplicate fetching. The distinction is about permitted use, not
necessarily the number of HTTP requests an operator will see.

OpenAI says a site blocked from OAI-SearchBot will not be shown in ChatGPT search answers, although navigational links
can still appear. Its documentation cautions that changes to `robots.txt` can take roughly 24 hours to affect search
systems. Those details make an audit more concrete: check the file served publicly, check the search bot's access, and
wait for the documented processing interval before treating a missing answer as evidence of a fault. Even then,
permission to appear is not a guarantee of a citation or a click.

Google's [crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
places `Google-Extended` in a different category again. It is a `robots.txt` product token, not a separate HTTP request
user agent. Google uses it to control whether content it crawls can train future Gemini models and be used for grounding
in Gemini Apps and Grounding with Google Search on Vertex AI. Google says `Google-Extended` does not affect inclusion or
ranking in Google Search. Its October 30 announcement adds a commercial complication: a site that specifically allows
`Google-Extended` remains eligible to appear in Gemini Apps responses and receive clicks through grounding. A site
considering an opt-out should therefore separate ordinary Google Search from Gemini grounding instead of treating both
as one visibility channel.

The documents expose two common mistakes. Blocking Googlebot to stop model training would change ordinary Google Search
access, even though Google provides `Google-Extended` for a different purpose. Blocking `OAI-SearchBot` to stop OpenAI
model training would change ChatGPT search eligibility, even though `GPTBot` is the training control in OpenAI's current
documentation. Neither mistake requires a sophisticated attack. It can arise when a team maintains one spreadsheet
column marked "AI bot" and maps every crawler to the same action. The new token helps only if the spreadsheet gains a
separate column for the use of fetched content.

Neither provider's documentation promises a publisher the same traffic after changing a rule. The decision is about
eligibility and permitted use; actual referrals depend on queries, product behavior, ranking and whether an answer sends
someone onward. A publisher that wants to preserve discovery should monitor search indexing, AI answer referrals and
training preferences as three separate series. Counting crawler requests alone would not resolve the question.

![An illustrated sorting gate sends a printed page toward an open catalog shelf while a second route stops before a closed vault.](/images/articles/shared-ai-training-opt-out-publishers-search/interior-v1.jpg)

_AI-generated editorial illustration. The routes represent separate search access and training-use decisions._

## The economic bargain remains unsettled

The new token has appeal because the old arrangement requires a growing list of named bots. The
[International Press Telecommunications Council's March guidance](https://iptc.org/news/iptc-publishes-v2-0-of-ai-opt-out-guidlines/)
describes several opt-out techniques already in use, including `robots.txt`, licensing and other rights signals. Its
advice is a reminder that publishers cannot simply rely on one syntax for every provider and every asset. A shared
signal could shrink maintenance work if adoption becomes broad. It cannot retroactively reconcile every older contract.

IPTC's updated guidance appeared seven months before the new individual draft. It added Really Simple Licensing and
Cloudflare's Content Signals to an already crowded set of techniques. That history matters to an editor maintaining a
live site: content has been published, crawled, syndicated and licensed under earlier rules. A new policy line applies
to the served file going forward according to the draft's model. It cannot tell a publisher which historical copies sit
in a model provider's archives or whether a syndication partner kept the rights metadata attached. The immediate work is
to identify what the publisher controls at its own host and what requires a conversation with partners.

Cloudflare has put a number on the mixed-use problem, with an important denominator. In a
[September 15 release](https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/),
it said crawlers that gather content for both search indexing and AI training represented **36.6% of verified crawler
traffic on its network**. It also said **fewer than 1%** of website owners blocked search crawlers while **17%**
restricted AI training. These are Cloudflare network and customer observations, not a census of the whole web. They show
why a binary block-or-allow choice is costly for publishers: many want discovery and a different training policy at the
same time.

Cloudflare's answer includes an [AI training control](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)
that it says can keep a site indexed while refusing training use by accountable mixed-purpose crawlers. It also launched
a [Pay Per Use beta](https://blog.cloudflare.com/pay-per-use/) in which an AI buyer proposes a price for a specified use
and reports uses for billing. That beta requires a buyer, an accepted offer and self-reported usage checked against
enrolled publishers. A shared training opt-out does none of those things. It states a preference against a use; it does
not establish a license, a price, a reporting obligation or a payment claim.

Cloudflare's **36.6%** measures verified crawler traffic on its network, not human visits, ad revenue or training
datasets. Its **17%** counts sites restricting training, not content excluded. The **under 1%** search-block figure is
another website choice. Plotting the three numbers as though they share a denominator would distort the story. Read with
their denominators attached, they show operators more willing to refuse training than to disappear from search.

Cloudflare's commercial proposal addresses a different hole in the new draft. In the beta, buyers specify the use they
will pay for and publishers accept an offer. Usage is reported by the buyer, then checked against enrolled publishers
before billing, according to Cloudflare's description. That is a transaction design with a price and a reporting path,
although its performance is not established by the launch post. The shared token has no field for a rate per article, a
definition of payable use in an answer, or a remedy if a buyer does not report one. A publisher seeking compensation
still needs a contract or payment system, and a review of its rights to sell that use.

A provider could interpret separate training and search controls precisely and continue sending people to useful pages.
A publisher could allow training for freely reusable documentation and restrict a paid archive. The value of that
arrangement would have to show up somewhere beyond the public file: in organic search clicks, AI referrals, crawler
access or licensing receipts. Those measures need a baseline and a review after the provider actually supports the
token.

The rights question is wider than `robots.txt`. The
[European Commission's guidance on general-purpose AI obligations](https://digital-strategy.ec.europa.eu/en/faqs/guidelines-obligations-general-purpose-ai-providers)
says model providers must maintain a policy to comply with Union copyright law and take measures to identify and respect
rights reservations. Google argues the new signal is consistent with the voluntary general-purpose AI Code of Practice.
That regulatory context explains why machine-readable declarations matter, but it does not turn this particular
individual draft into a legal command or decide a dispute about an individual work. Contracts, jurisdiction and the use
of a given asset still matter.

## Sort rights by page before writing rules

Begin with the work being published, not the name of the crawler. A newsroom may control its original articles and
merely display licensed images. It may publish a press release freely but reserve a member archive. Each class asks for
a separate rights decision. The table below is an operating worksheet, **not a universal robots.txt configuration**.
Each rule needs a rights review and a live test against the provider's current documentation.

| Question for a page class                                 | Present control or record                                         | What the new token could add                                                                     | What the result cannot prove                               |
| --------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- |
| Should Google Search index this page?                     | Googlebot access, index directives, sitemap and Search Console    | Nothing directly; the draft concerns training use                                                | A training rule does not guarantee ranking or clicks       |
| Should ChatGPT search surface it?                         | OAI-SearchBot access and referral measurement                     | Nothing directly; OpenAI keeps current crawler controls                                          | An allowed search bot does not guarantee citation          |
| May OpenAI use it for model training?                     | GPTBot rule, contract and content-rights inventory                | A common use preference once OpenAI implements it                                                | A declaration alone does not show dataset removal          |
| May Google use it for model training or Gemini grounding? | Google-Extended, explicit rights agreements and product reporting | Google plans a default interpretation in 2027 when no specific Google-Extended choice is present | A Google Search result does not show Gemini use or non-use |
| May another AI buyer pay for a defined use?               | Negotiated or platform licensing terms and usage receipts         | No price term; the token can inform a separate negotiation                                       | An opt-out is not a payment mechanism                      |

The table separates **crawl access**, **content use**, **appearance** and **compensation**. Those are different fields
in an operator's decision, even when one provider collects a page only once. The new token makes one field easier to
communicate. It does not merge the others into a complete contract.

A rights inventory also prevents an accidental overstatement. Who owns a commissioned photo? Is a wire-service paragraph
licensed only for display? Do contributors retain training rights? Does a paywall cover the same asset as the public
excerpt? If the answers differ, the site's path-level rules and contracts need to reflect the differences. A technical
rule written at the root of a site can be broader than the rights the site holds.

Consider an **illustrative scenario, not a reported meeting or customer case**. A regional publisher places original
stories at `/news/`, agency photos under `/images/`, public event notices at `/events/` and paid data at `/research/`.
The editor wants the news articles visible in Google Search and ChatGPT search. The photo agreement permits display but
does not spell out model training. The research subscription contract reserves reuse. One root-level `Disallow` for
every AI-associated bot would be easy to type and hard to defend: it might close search routes the editor wants, while
saying nothing precise about the photo contract. A purpose group with path rules could state a narrower training
preference once participating providers support it. The team would still need separate crawler access rules and the
rights owner's approval for the photo and data classes.

There is no public measurement that can tell this hypothetical publisher the perfect rule beforehand. If the search team
sees few AI referrals today, it may make a different business choice from a site receiving many qualified subscribers
from answer engines. If a provider offers a paid training license, the contract's value depends on the covered material,
term, reporting and enforcement, none of which fit in a `User-Agent` line. A single default across all publishers would
hide those differences. The matrix keeps them visible at the decision meeting.

## What can be verified now

An operator can perform several checks without waiting for 2027. First, record the current public `robots.txt` response
for each hostname and verify that it is reachable. Second, list the crawler-specific rules already in force, including
`GPTBot`, `OAI-SearchBot`, Googlebot and `Google-Extended`, and compare them with the site's stated editorial and
commercial policy. Third, identify the path classes whose rights differ. Fourth, capture current search and referral
baselines with the same properties and windows, and preserve any license receipts separately. These are present-tense
checks, not proof of future compliance.

The new token can then be evaluated as a proposed addition. The draft says an explicit policy group is required; a
wildcard group does not carry the new preference. The file should be tested for the paths it means to cover. A
provider's published implementation note should be checked before the site treats the token as operative for that
provider. If a company-specific rule is intended to override the shared choice, document that choice explicitly and
recheck the provider's precedence rules. The draft says specific expressions should generally take precedence and
private agreements may override preferences, but the service operator still has to know what it agreed to.

The last check is an outcome check with honest limits. Server logs can show requests from identifiable crawlers. Search
Console can show search indexing, impressions and clicks for the canonical property. Analytics can show referred visits
when tracking is present. A licensing ledger can show agreed payments. None of these sources can fully observe a model
developer's internal training set or prove that a particular paragraph never influenced a model. An operator should
label that evidence gap instead of filling it with a crawler count.

A preference file may be changed on Tuesday, while a provider's support begins months later and a search index updates
on its own schedule. A fall in visits on Wednesday would not establish that the training rule caused it. A rise in
referrals would not establish that the provider honored a training restriction. Keep the dates of the file change,
provider implementation, crawl observations and traffic windows next to each other. Then a publisher can ask a narrower
question: did the search route remain eligible after the policy change, and did the provider say it interpreted the
training signal for the relevant paths? That is a checkable operating result, even though it falls short of an audit of
model weights.

The publisher's next `robots.txt` review should therefore have two dates beside it: October 30 for the announcements,
and a blank space for each provider's actual implementation. Until those spaces are filled, the current bot rules and
contracts carry the decision.

## Continue reading

- [Perplexity AI Search: Product Model, Publisher Economics, and Unresolved Legal Risk](https://digidai.github.io/2026/02/12/perplexity-ai-search-engine-reinvention-deep-analysis/): Trace the answer-engine business model and the older publisher payment dispute.
- [Aravind Srinivas and Perplexity: Building an Answer Engine](https://digidai.github.io/2025/11/08/aravind-srinivas-perplexity-deep-analysis/): Read how an answer engine built its product around retrieving other sites' work.
- [ChatGPT Ads Reach 89 Countries, but Auction Wins Are Not Impressions](https://digidai.github.io/2026/10/22/chatgpt-ads-89-countries-impression-measurement/): Separate a platform's announced inventory from measured impressions and publisher value.
- [Google DeepMind After the 2023 Merger: Verified Outputs, Integration, and Open Questions](https://digidai.github.io/2026/03/06/google-deepmind-real-fighting-power-two-years-after-merger/)