Local AI Advisor
All posts

One robots.txt line to opt out of AI training

OpenAI, Google, Microsoft, and Anthropic now back one shared robots.txt token for opting a site out of AI training.

By Ray, Your Local AI Advisor6 min readai-newsbeginners

Until now, opting your website out of AI training meant maintaining a separate rule for every company's crawler: one name for OpenAI, another for Google, another for Anthropic, and more as new crawlers appeared. On October 30, 2026, that changed. OpenAI, Google, Microsoft, and Anthropic said they will honor a single shared signal in robots.txt, so one entry can express your training preference to all four at once. If you run a blog, a docs site, or a small business page, this is the simplest control of its kind so far — with a few limits worth knowing before you rely on it.

A stack of web pages beside a glowing gate, with a friendly robot waiting on the other side

What happened

The signal comes from an Internet-Draft titled "Machine-Readable Training Preference Signal," published October 30, 2026 and written by Kevin Kelley of Anthropic for the Internet Engineering Task Force's AI Preferences (AIPREF) working group. It introduces one policy token, __usage_ai-training, that sits in the User-Agent line of an ordinary robots.txt group. It adds no new directives and no new grammar: it reuses the path rules, longest-match precedence, and group structure that robots.txt already has.

Google confirmed its side the same day on the Search Central blog, saying it plans to support the shared control in 2027 alongside other AI providers. The other companies' timelines differ: this is a shared promise, not a switch that flips everywhere at once.

The two lines

A site that wants to opt out of AI training writes this in its robots.txt:

User-Agent: __usage_ai-training
Disallow: /

That says: for every path on the site, the preference is against AI training. You can also carve out exceptions with Allow lines. The draft's own example opts everything out except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Two details in the draft are easy to get wrong. First, a catch-all group (User-Agent: *) does not count: the draft says wildcard groups must not be interpreted as matching the policy token, so if your file has no __usage_ai-training group at all, no training preference has been expressed — whatever your other rules say. Second, company-specific rules still work alongside the shared one, and the draft says specific expressions should take precedence over general ones. Google confirmed this behavior: if a site opts out with the shared token but does not mention Google-Extended, Google plans to treat that as a Google-Extended disallow, while sites that explicitly allow Google-Extended stay eligible to appear in Gemini Apps responses through grounding.

What it does not do

This is the part most people will misunderstand, so be precise. The token covers training only. The draft states plainly that it is not a directive for crawling behavior: it does not tell a crawler it cannot fetch your pages, and it does not keep your pages out of search results or AI-written answers that quote them. If your goal is to disappear from AI answers, this signal is not the tool for that job. Microsoft, in fact, is adding a separate "AI Grounding" setting for whether a page can feed AI answers in Bing, Copilot, and Microsoft 365, according to same-day reporting — a sign that training and answering are treated as two different permissions.

Diagram of web pages passing through a gate: one path is blocked, while another path continues to a search symbol

It also does not reach backward. Updating robots.txt today does not remove content already collected into existing datasets or already baked into trained models. It is a preference for future training runs, expressed by the companies that choose to honor it — and the draft itself is an individual Internet-Draft, not an endorsed IETF standard. It expires May 3, 2027 unless it is updated, and it only binds the companies that adopt it.

When each company starts honoring it

The timelines are uneven:

  • Anthropic has already begun implementing support, according to same-day reporting. It also wrote the draft.
  • OpenAI says it aims to implement the signal in the coming months, and its existing crawler controls keep working alongside it.
  • Microsoft targets February 1, 2027, and is going further than the shared token with page-level training and grounding settings. Sites still relying on Bing's old NoArchive and NoCache tags have until March 1, 2027 to switch, according to same-day reporting.
  • Google says it aims to support the control in 2027, with its crawler documentation pointing to early 2027.

If you add the token now, you are planting a flag early. The protection only becomes real company by company, on each company's schedule.

What this changes for you

If you publish on the web — a blog, docs, a portfolio, a shop — this is the first time one line covers four major AI builders at once, instead of a growing list of per-company crawler names. Adding it now costs nothing, and removing the group later removes the preference just as easily. That reversibility is a real advantage over blocking crawlers outright, which can also cost you search visibility.

If you run AI locally, the connection is quieter but real. A model running on your own machine never sends your documents to a training pipeline in the first place — that is the privacy case for local AI, and it does not depend on any company's promise. What this signal changes is the other direction: the public web content that future models get trained on. As more sites opt out, tomorrow's training corpora will look different from today's. None of that changes what your local model can do today, but it is one more reason to keep your own data local rather than assuming the public web will stay an open buffet.

For the earlier, blunter version of this story — sites blocking AI agents entirely — see the October 18 post on the web shutting its doors to AI agents.

Honest limits

Three cautions before you treat this as settled. First, the announcement names four companies. Other AI builders, scrapers, and dataset collectors are not covered unless they adopt the token too, and a voluntary signal is only as strong as its adopters. Second, the draft is still a draft: it could change before it becomes a standard, and its May 2027 expiry means the working group has to keep it alive. Third, nothing here guarantees enforcement. A robots.txt preference has always been a request that well-behaved crawlers honor, not a technical lock.

The bottom line

One shared token, four major companies, two lines in a file you already have. If you do not want your site's content used to train AI models, add the __usage_ai-training group now and note the timelines: Anthropic first, Microsoft by February 1, 2027, Google and OpenAI during 2027. If you want to stay out of AI answers too, or keep your search presence exactly as it is, you will need the separate, company-specific controls — this token alone does not do those jobs. And if you want the strongest version of "my content stays mine," running models locally remains the option that needs no one's permission at all.

Sources