The four biggest AI labs agreed on a single robots.txt signal to block AI training on your website content. Here is what it means and what service businesses should do today.
Ido Cohen · Published 2026-10-30 · SEO & Search
The four biggest AI companies in the world just agreed on one line of code that lets you tell all of them — simultaneously — to stop training their models on your website's content. On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic each announced they will honor a shared robots.txt policy token called __usage_ai-training, ending years of a fragmented, provider-by-provider opt-out system. If you own a service business website — a law firm, dental practice, HVAC company, med spa, or anything in between — this decision directly affects whether your blog posts, service pages, and customer-facing copy end up inside the next generation of AI models without your permission.
Before today, opting your website out of AI training was a full-time job in itself. You had to identify each AI company's individual crawler bot, then write a separate robots.txt rule for each one — GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and so on. Miss one and your content was fair game.
According to OpenAI's official announcement, they are now committing to honor a shared signal in robots.txt — a single file that websites already use to communicate preferences to crawlers — so site operators can express one AI training preference that reaches multiple participating providers at once. Google's Search Central Blog confirmed the same on October 30, calling it a plan to support "a shared AI training control in robots.txt" based on work from the AIPREF working group at the Internet Engineering Task Force (IETF), the body that sets the technical standards underpinning the internet.
The new token is __usage_ai-training. Think of it as a universal "do not train on my site" label that all four major labs have pledged to read and respect.
The timeline matters:
This is not a done deal — it is a voluntary commitment based on an IETF Internet-Draft, not a ratified standard. But when OpenAI, Google, Microsoft, and Anthropic move in the same direction on the same day, that is the clearest signal the industry has sent yet that this opt-out mechanism is becoming the de facto standard.
Most service business owners are not thinking about this. That is a mistake.
Here is the practical issue: every blog post, FAQ page, before-and-after gallery description, pricing guide, and "About Us" story you have published is publicly crawlable. AI companies have been ingesting that content to train large language models (LLMs) — the AI brains behind ChatGPT, Gemini, Claude, and Copilot. That training data shapes how AI systems answer questions about plumbers, dentists, real estate agents, and contractors.
When a homeowner asks ChatGPT "what should a plumbing inspection cost?" or "how do I pick a personal injury lawyer?", the answer is built in part from content scraped from service business websites — possibly yours. You wrote it. You paid your content writer for it. And until now, opting out meant managing a growing list of individual bot rules that most small business owners had no idea existed.
There is a second angle here that is even more important: differentiation. The content your business produces — your specific tone, your local expertise, your proprietary service descriptions — is a competitive asset. If that content gets absorbed into a generic AI model, it stops being an edge. Your competitor's plumbing blog and yours become undifferentiated training data that gets averaged into the same AI response.
A robots.txt file is a plain text file that lives at the root of your website (e.g., yourbusiness.com/robots.txt). It tells automated bots — search engine crawlers, AI scrapers, and others — what they can and cannot access. Every website has one (or should).
Under the old system, blocking AI training meant listing individual crawlers by name. According to Unite.ai's reporting, the new Internet-Draft authored by Anthropic's Kevin Kelley defines a cross-vendor policy token, __usage_ai-training, that functions as a single universal switch. The example in the draft pairs the line User-Agent: __usage_ai-training with Disallow: / to express a preference against AI training for all paths on a site — but you can also allow specific folders like a press kit while blocking everything else.
Critically, according to Google's Search Central Blog, when a website uses this policy token to opt out of training, Google plans to interpret it as if the publisher had disallowed Google-Extended — Google's existing Gemini training control — even if Google-Extended is not explicitly listed. That means one line does the work of multiple separate rules across all four providers.
One important nuance from Unite.ai's coverage: this token is a preference signal, not a crawling directive. Blocking AI training does not block search indexing. You can still show up in Google Search, Bing, and AI-powered search results. You are only controlling whether your content is used to train the underlying models, not whether it is cited or surfaced in answers.
Before you add a single line of code and call it a day, understand what this opt-out cannot do:
1. It is not retroactive. Changes to robots.txt apply only after a crawler retrieves the updated file and then re-fetches the affected pages. Content already ingested into training datasets stays ingested. This opt-out is about the future, not the past.
2. It only covers compliant crawlers. Major labs — OpenAI, Anthropic, Google, Microsoft — have stated they will honor this signal. ByteDance (TikTok's parent) and many smaller AI companies are a different story. For non-compliant bots, you need firewall-level rules, not just robots.txt.
3. The standard is not final. This is an IETF Internet-Draft, not a ratified RFC. The draft expires May 3, 2027. Voluntary commitments from four companies are meaningful, but "voluntary" is still the operative word.
4. Opting out of training ≠ opting out of AI citations. If Google AI Overviews or ChatGPT Search cites your website in an answer, that happens through different crawlers (search/retrieval bots) than training crawlers. Blocking training does not block citations — and you likely do not want to block citations.
This is the real decision, and there is no universal right answer. Here is a framework for service businesses:
The default for most service businesses with even a modest content library: opt out of AI model training while keeping all search crawlers active. You preserve your competitive content moat and still get found in AI-powered search results. You just are not feeding the model that your competitor's ChatGPT queries are running on.
This is not a decision that requires months of deliberation. Here are concrete steps you can take right now:
1. Find your robots.txt file. Type yourdomain.com/robots.txt into your browser. If it is empty or returns a 404, your site has no robots.txt — fix that first with your web developer or via your CMS (WordPress, Squarespace, Wix all support robots.txt editing).
2. Add the new unified token. Once the signal is live (Anthropic already supports it; others roll out through 2027), add the following block to your robots.txt to block AI training across all participating providers:
User-Agent: __usage_ai-training
Disallow: /
This does not block Google Search, Bing, or any retrieval bot. It specifically targets the training preference signal.
3. Keep your existing search crawlers allowed. Make sure your Googlebot, Bingbot, and other search crawler rules are still set to Allow: /. Do not accidentally block the bots that index you for organic search.
4. Block training-specific crawlers individually now — do not wait. The new unified token is rolling out through 2027. In the meantime, add individual rules for GPTBot (OpenAI training), ClaudeBot (Anthropic training), and Google-Extended (Gemini training) separately, so you have coverage today.
5. Talk to your web developer or marketing agency about this. Show them this post. Ask them to audit your robots.txt and confirm what you are and are not blocking. This is a 15-minute conversation that most service businesses have never had.
6. Set a calendar reminder for February 1, 2027. That is when Microsoft's Copilot and Bing AI commit to honoring the new signal. Worth a re-audit at that point.
---
Will blocking AI training hurt my Google rankings?
No. The __usage_ai-training token and the individual training bot rules (GPTBot, ClaudeBot, Google-Extended) are separate from Google's search crawlers like Googlebot. According to Google's own Search Central Blog, opting out of AI training does not affect search indexing. Your pages can still rank, still appear in AI Overviews, and still be cited by AI search tools — you are only controlling whether your content trains the underlying models.
What is robots.txt, and do I already have one?
A robots.txt file is a text file at the root of your website that tells bots what they can and cannot crawl. Almost every website has one automatically generated by their CMS. Type yourdomain.com/robots.txt into a browser to check. If you see a page of text with "User-agent" rules, you have one. If you get a 404 error, you need to create one — any web developer can do it in minutes.
If I opt out, can AI tools still mention my business in answers?
Yes. AI search tools like Google AI Mode, ChatGPT Search, and Perplexity use separate retrieval crawlers to find and cite sources in real time — these are different from the training crawlers you are blocking. Opting out of training does not prevent your site from being cited or linked in AI-generated answers. Think of it as controlling the recipe vault (training data), not the delivery window (search citations).
Does this apply retroactively — can I get my content removed from existing AI models?
No. Robots.txt changes and the new opt-out signal only apply to future crawls. Content that has already been ingested into training datasets is not affected by this update. If you want to pursue content removal from existing models, you would need to contact each AI company individually — a slow and uncertain process. This is a forward-looking control only.
What if I'm not sure whether to opt in or out?
The conservative default for most service businesses with substantive content (detailed blog posts, how-to guides, service explanations) is to opt out of AI model training while keeping search indexing fully enabled. If you have thin or template-heavy content and very little differentiated IP on your site, the risk of staying opted in is lower. When in doubt, consult your digital marketing agency about your specific content portfolio before making a change.
---
Sources: