Should you block AI crawlers? What each bot does
Published 8 min read
Bot facts from each vendor's own docs, checked on 11.11.2026.
Should you block AI crawlers like GPTBot? It depends on which ones. Blocking a training bot and blocking a search bot are two different choices, with different results.
Card needed. Cancel before day 7 and nothing is charged.
Training
GPTBot and the Google-Extended token. Used to train models.
Search
OAI-SearchBot, PerplexityBot, Googlebot. Feeds AI answers now.
User fetch
ChatGPT-User, Perplexity-User. Opens a page when a person asks.
The short answer
Blocking AI training bots and blocking AI search bots are two different choices. Block GPTBot, and OpenAI should not train on your pages. Block OAI-SearchBot, and ChatGPT search cannot read your pages when it answers.
Bot facts from each vendor's own docs, checked on 11.11.2026.
Key points
GPTBot is not ChatGPT search. OpenAI says each bot setting is independent of the others.
Google-Extended is not Googlebot. Google says Google-Extended does not affect your place in Google Search.
Check your CDN. Since 15 September 2026, new Cloudflare domains block some AI bots by default on pages with ads.
What does each AI bot do, and what does blocking it mean?
Each AI company sends more than one bot. The table lists the bots of ChatGPT, Perplexity and Google, in each company's own words.
| Bot (robots.txt name) | Company | Job | If you block it | Source |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Your pages should not be used to train OpenAI's models. ChatGPT search is a separate setting. | OpenAI |
| OAI-SearchBot | OpenAI | Search | Your site is not shown in ChatGPT search answers. It can still appear as a plain navigation link. | OpenAI |
| ChatGPT-User | OpenAI | User fetch | OpenAI says robots.txt rules may not apply, because a person asked for the page. | OpenAI |
| PerplexityBot | Perplexity | Search | Perplexity cannot use it to show and link your site. Perplexity says it is not used for training. | Perplexity |
| Perplexity-User | Perplexity | User fetch | Perplexity says it generally ignores robots.txt, because a person asked. | Perplexity |
| Googlebot | Search, also used for AI Overviews | You leave Google Search, and with it AI Overviews. | ||
| Google-Extended | Gemini training and grounding | Gemini stops using your pages for training and for grounding in Gemini Apps. Google Search stays the same. |
Each row from the company's own crawler page, read on 11.11.2026. Google-Extended is a robots.txt name only; Google crawls with its normal bots.
Does blocking GPTBot remove you from AI answers?
Not from ChatGPT search. OpenAI says each bot setting is independent, so you can block GPTBot and still allow OAI-SearchBot.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /- If you allow both, OpenAI says it may use one crawl for both jobs.
- OpenAI says a robots.txt change takes about 24 hours to reach ChatGPT search. More on timing: how often AI updates its information.
- What a model learned before your block stays in that model.
Google-Extended vs Googlebot: which one controls AI Overviews?
Googlebot does. AI Overviews are part of Google Search, and Google says Google-Extended does not affect inclusion in Google Search.
- Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI.
- To limit what AI Overviews show from a page, Google points to nosnippet, max-snippet and noindex. Those limit normal search results too.
What changed for AI bots on Cloudflare on 15 September 2026?
Cloudflare now lets site owners treat Search, Agent and Training bots apart. Since 15 September 2026, new domains block Training and Agent bots by default on pages that show ads.
Search
Collects or indexes your content to answer questions later. Allowed by default.
Agent
Acts for a person in real time, such as a chat fetch bot.
Training
Takes your content to train or fine-tune a model.
- Cloudflare judges a bot with several jobs by all of them. Its post names Googlebot, Applebot and BingBot. Customers who block Training also block those bots.
- So blocking Training on Cloudflare can close your site to Google Search too. Check that setting first.
- Customers could mark in their Security settings, before 15 September, that they want no change for these mixed bots.
Reports in the press differ on who gets which default. We quote Cloudflare's own post of 1 July 2026 only.
When does blocking an AI bot make sense?
Block training bots if you do not want your text in future models. Keep search bots open if you want AI answers to read and link your pages.
Choices that fit together
Do: Want AI search to read you: allow OAI-SearchBot, PerplexityBot and Googlebot.
Do: Want no training: block GPTBot and Google-Extended, keep the search bots.
Do: Worried about load: ask your host about rate limits for one bot, not a full block.
Choices that often backfire
Avoid: Blocking "User-agent: *" to stop one bot. It stops every bot without its own group.
Avoid: Blocking Training on a CDN without checking what happens to Googlebot.
Avoid: Trusting robots.txt alone to stop user fetch bots. Some of them may not follow it.
- There is no right answer for every company. It is a choice between reach in AI answers and control over training.
To see what your site blocks today, run the 4 checks for AI crawlers.
What is pay per crawl?
Pay per crawl is a Cloudflare feature that lets a site charge AI crawlers for access. It is in closed beta.
- A crawler that agrees to pay gets the page. Others get an "HTTP 402 Payment Required" answer with the price.
- Crawler and site owners can ask to join with a form. Existing Enterprise customers ask their account team.
How does Nenn show which AI bots can read your site?
The Website check in the trial lists which AI bots your robots.txt and firewall let in. Failed checks come first.
AI bots row
Per bot: allowed or blocked, from robots.txt and from what the site answers.
Fix with Nenn
Nenn drafts the robots.txt change. You approve it before anything changes.
Then the weekly check
See if ChatGPT, Perplexity and Google AI Overviews name you after the change.
Nenn reads your site with its own bot. See how NennBot works.
Questions about blocking AI bots
Should I block GPTBot?
Only if you do not want OpenAI to train on your pages. Blocking GPTBot does not remove you from ChatGPT search. That is OAI-SearchBot, a separate setting.
How do I block GPTBot in robots.txt?
Add two lines: "User-agent: GPTBot" and "Disallow: /". Leave OAI-SearchBot allowed if you want ChatGPT search to read your site.
Is robots.txt enough to keep AI bots out?
Not always. OpenAI and Perplexity say their user fetch bots may not follow robots.txt, because a person asked. A firewall can block them, but then that person cannot open your page through AI either.
Does Cloudflare block AI crawlers by default?
For new domains since 15 September 2026, Cloudflare blocks Training and Agent bots on pages that show ads. Search bots stay allowed. Check your own dashboard for older domains.
Do I need llms.txt if I allow AI bots?
No. Google says its AI features need no special files. An llms.txt file does no harm, but it does not open or close your site to any bot.
Not sure what your CDN blocks?
The Website check in the trial lists which AI bots can read your site. Then the weekly check shows if AI names you.
Card needed. Cancel before day 7 and nothing is charged.
Which AI bots can read my website?
- 1Your site?
Keep reading
From the bot choice to checking what your site does today.