TL;DR
Can one line in robots.txt stop AI companies from training on your website?
OpenAI, Google, Microsoft and Anthropic say they will respect one shared robots.txt line. It tells them not to train AI on your site. It's a polite request, and showing up in AI answers is a separate choice. Nothing is live yet, so pick the setup that fits you before 2027.
What is the new AI training opt-out line in robots.txt?
Robots.txt is a small text file at the root of your website (yoursite.com/robots.txt) that tells automated visitors what you'd like them to do. Until now, keeping your writing out of AI training meant blocking each company's crawler by name: GPTBot, Google-Extended, ClaudeBot and a list that kept growing.
On October 30, 2026, OpenAI, Google and Microsoft's Bing each said they will honor one shared "policy token," a special name you put on the User-Agent line. It comes from an IETF draft (the IETF is the group that writes many internet standards) by Anthropic engineer K. Kelley, and The Verge reports Anthropic has adopted it too. The draft's own example looks like this:
`` User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ``
That says "please don't train AI on anything here, except the press kit folder." Note the two underscores at the start of __usage_ai-training. Some write-ups drop them, and a typo means the line does nothing.
When does the AI training opt-out start working?
Not yet. OpenAI says it aims to implement the signal "in the coming months." Google says it aims to support it "in 2027." Bing gives the firmest date: it starts honoring the new preferences on February 1, 2027, as it crawls your pages.
So there's no rush. You can add the line today and it will sit quietly until each company switches it on. Adding it early also means you won't forget.
Does blocking AI training remove you from AI answers?
Not automatically, and this is the choice that matters most. AI companies use your pages in two different ways:
- Training: your text helps build the next AI model. You don't get a link or a visit out of it.
- Grounding: an AI tool reads your page while answering someone's question, then cites and links to you. This is where AI referral traffic comes from.
The new line only covers training. Bing says outright that its training opt-out does not affect your eligibility for normal Bing search results, and it is adding a separate, page-level "AI Grounding" setting. Google is less tidy. Its long-standing Google-Extended control covers both training for Gemini and appearing in Gemini Apps answers. Google says the new line counts as blocking Google-Extended unless you set Google-Extended yourself, and that sites which explicitly allow Google-Extended stay eligible for cited, clickable answers in Gemini Apps.
Training uses your work. Grounding sends you visitors. Decide on each one separately.
Which robots.txt setup should you use?
Most small sites fit one of three setups.
1. "Train on me, cite me." Do nothing. With no line in your file, no preference is expressed, and Bing says it treats that as allowed.
2. "Don't train on me, but show me in AI answers." Add the training line. Then decide on Google: if Gemini Apps traffic matters to you, explicitly allow Google-Extended, knowing that also says yes to Google using your pages for Gemini.
``` User-Agent: __usage_ai-training Disallow: /
User-Agent: Google-Extended Allow: / ```
Leave the Google-Extended part out if you'd rather Google skip you for both. On Bing, grounding stays on by default, so there's nothing more to add.
3. "Keep me out of AI as much as possible." Add the training line, disallow Google-Extended, and block the AI crawlers you care about by name. For Bing, add its page-level grounding tag to each page, <meta name="Bingbot" content="ai-grounding:disallow">. Bing warns this "will significantly reduce" your visibility in AI answers, and neither setting changes your normal Bing search listing.
Two details trip people up. A User-Agent: * rule does not count as the training line; the draft says wildcard groups must not be read that way, so you need the exact token. And your existing rules for named crawlers keep working alongside it.
Can AI companies ignore robots.txt?
Yes. Robots.txt has always been a request, not a lock. The new line asks the companies that signed on to respect your choice. A company that didn't sign on, or a scraper that never asked, can still read your public pages. Meta voiced support, per The Verge, but hasn't committed.
The line also comes from an individual IETF draft, which can still change before it becomes a standard. And a "no" today doesn't pull your pages out of models that were already trained.
That's why we'd keep expectations modest. The line is still worth a minute, because the biggest AI labs have now said in writing they'll honor it. On Reddit, a thread titled "How to Keep Your Writing Indexed by Google (But Opt Out of AI Training — As Much as Possible in 2026)" shows how many writers want exactly that middle ground.
What should you do now?
Open yoursite.com/robots.txt and see what's there. On WordPress, Wix, Squarespace and similar builders, the file is often managed in settings or through an SEO plugin, so look there before editing anything by hand.
Then pick your setup from the three above, write it down, and add it whenever you're ready. If you blocked AI crawlers by name in the past, keep those rules. Bing users who rely on the old NoArchive or NoCache tags have extra homework: Bing says those stop counting for training and grounding after March 1, 2027.
Our view: most bloggers and small businesses want setup 2. AI answers that cite you can send real visitors, and training gives you nothing back. Copy the token exactly, check that it saved, and stop guessing at crawler names.
FAQ
Is it __usage_ai-training or usage_ai-training?
Two leading underscores: __usage_ai-training. The draft says the prefix makes accidental matches unlikely, so the version without them won't be read as the shared signal.
Can I block AI training on just one folder?
Yes. Put that folder in the Disallow line, like Disallow: /recipes/, or allow one folder inside a full block, as the draft's press-kit example does. Bing also lets you set training per page with a meta tag.
Will this line hurt my Google or Bing rankings?
Bing says its training preference doesn't change eligibility for regular Bing results. The draft says the token only expresses a preference about how content is used and sets no crawling rules, so it isn't meant to stop search engines reading your pages.
Does it remove my site from AI models that already exist?
No. It's a preference for future training. Nothing in these announcements covers removing content from models already built.
Sources
- IETF draft: Machine-Readable Training Preference Signal (K. Kelley, Anthropic), October 30, 2026; the token, example and wildcard rule.
- Google Search Central: Supporting a shared AI training control for websites, October 30, 2026; Google-Extended and the 2027 target.
- Bing: Your content, your choice, October 30, 2026; Feb 1 and March 1, 2027 dates, grounding tags. Also Bing Webmaster help: new AI preferences.
- OpenAI: Making it easier for websites to express AI training preferences, October 30, 2026.
- The Verge on the shared opt-out, Anthropic's adoption and Meta's response.
- Reddit r/SEO_AEO_GEO thread, title only, found through Google search.
