Can AI see your website? 4 checks for AI crawlers
Published Updated 10 min read
Checks 1 to 3 run by hand on nenn.io on 11.11.2026 with curl and the page source. Not a run of the Nenn app.
Can ChatGPT see your website? It can if its bots may fetch your pages and your text sits in the HTML. Here are 4 checks, in the order a bot meets them.
Card needed. Cancel before day 7 and nothing is charged.
User-agent: OAI-SearchBotDisallow: /User-agent: PerplexityBotAllow: /User-agent: GooglebotAllow: /
Scheme: one robots.txt group stops OAI-SearchBot at the gate. PerplexityBot and Googlebot reach the site.
The short answer
ChatGPT can use your pages only if its bots may fetch them and the text is in the HTML. Four gates decide it: robots.txt, firewall, text without JavaScript and the search index. Readable is still not named.
Checks 1 to 3 run by hand on nenn.io on 11.11.2026 with curl and the page source. Not a run of the Nenn app.
Key points
Search bot, not training bot. For ChatGPT answers, OAI-SearchBot must be allowed. GPTBot is for training.
Two places can block. robots.txt can say yes while your firewall says no. Check both.
Readable is not named. Passing the 4 checks lets AI read you. It does not make AI name you.
Check 1: Does your robots.txt let AI bots in?
Open yoursite.com/robots.txt in your browser. If a group for OAI-SearchBot, PerplexityBot or Googlebot says "Disallow: /", that engine cannot read your site for its answers.
What to look for
Your engine's bot by name
OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, Googlebot for Google AI Overviews.
The star group
"User-agent: *" with "Disallow: /" blocks every bot that has no group of its own.
Old plugin rules
Some SEO and security plugins write bot rules for you. Read the live file, not the settings screen.
- OpenAI says a robots.txt change takes about 24 hours to reach ChatGPT search. More on timing: how often AI updates its information.
- ChatGPT-User is different. It fetches a page when a person asks for it, and OpenAI says robots.txt rules may not apply to it.
Training bots and search bots are two separate choices. See what each AI bot does before you block one.
Check 2: Does your firewall or CDN stop AI bots?
Your robots.txt can allow a bot while your firewall blocks it. A blocked bot gets an error page or a challenge, not your text.
How to check yours
Ask one question
Ask your host or developer: do we block AI bots, unknown bots or bots without JavaScript?
Open the bot settings
On a CDN like Cloudflare, look at the AI bot and bot fight settings for your domain.
Read the logs
Search your server logs for OAI-SearchBot or PerplexityBot. A 403 or 503 next to the name means a block.
- Cloudflare split its AI bot control into Search, Training and Agent bots. Since 15 September 2026, new domains on Cloudflare block Training and Agent bots by default on pages that show ads.
- Agent bots include fetch bots like ChatGPT-User. So a page with ads may stay closed when a person pastes your link into a chat.
The full change, and who it affects, is in should you block AI crawlers.
Check 3: Is your text in the HTML without JavaScript?
Right-click your page and choose "View page source". If your main text and prices are not in that source, most AI bots will not see them.
How to check yours
View the source
Press Ctrl+U (Cmd+Option+U on a Mac). Search the source for one sentence from the page.
Find your price
Search the source for your price. Not there? It is drawn by JavaScript or is an image.
Check tabs and sliders
Text that loads only after a click may be missing for bots too.
- In a December 2024 study, Vercel found that none of the major AI crawlers rendered JavaScript. That included OAI-SearchBot, ChatGPT-User, GPTBot and PerplexityBot.
- Google is different. Googlebot renders JavaScript, and Google AI Overviews use pages from Google Search.
- Bots can change. Still, text in the HTML is the safe choice for every engine.
Readable: <p>Basic plan: 29 per month</p>
Not readable: <div id="price"></div> (filled by JavaScript later)Check 4: Can AI search find the page in an index?
AI search often starts with a web search. A page that no search index holds is hard for AI to find, even when bots may read it.
How to check yours
Search for the page
Type site:yoursite.com/pricing into Google and into Bing. No result means the page is probably not indexed.
Use the free tools
Google Search Console and Bing Webmaster Tools show if a page is indexed and why not.
Look for noindex
In the page source, search for "noindex". It keeps a page out of search, and so out of AI Overviews.
- Google says a page must be indexed and able to show a snippet to appear in AI Overviews. There are no extra technical rules.
- OpenAI says ChatGPT search rewrites a question into searches and may send them to outside search providers.
The 4 checks on one card
Run them in this order. The first gate that fails is the one to fix first.
| Check | How | Pass |
|---|---|---|
| robots.txt | Open /robots.txt | No "Disallow: /" for OAI-SearchBot, PerplexityBot, Googlebot or * |
| Firewall | Ask your host, read the logs | Bots get HTTP 200, not 403 or a challenge |
| HTML text | View page source | Main text and prices are in the source |
| Index | site: search, Search Console, Bing Webmaster Tools | Page is indexed and has no noindex |
What did the 4 checks show on our own site?
We ran the 4 checks on nenn.io on 11 November 2026, and checks 1 to 3 passed. Check 4 comes from Google Search Console and Bing Webmaster Tools.
| Check | What we saw |
|---|---|
| robots.txt | 15 AI bot names are allowed, among them OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot and Perplexity-User. Only /api/ is closed. |
| Firewall | The site runs behind Cloudflare. /pricing answered HTTP 200 to the user agents of GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and Googlebot. |
| HTML text | The source of /pricing held about 950 words before any JavaScript ran. Plan names and prices were plain text. |
| Index | Google Search Console, 12.11.2026: /pricing was not in Google's index yet. Bing not checked yet. |
If AI can read your site, will it name you?
Not by itself. What your pages say, and who else mentions you, decide if AI names you.
- AI names a company when a page answers the exact question in plain words. A home page rarely does that.
- In our own runs, pricing pages got the most citations. See which pages AI cited.
- Other websites matter too. Lists, reviews and directories that mention a rival can tip the answer.
The other 4 reasons AI names a rival are in why AI recommends my competitor.
How does Nenn run these checks for you?
The Website check in the trial tests what AI crawlers can read: robots.txt, firewall and page text. It takes about a minute and uses no credits.
AI bots
Which AI bots your robots.txt and firewall let in.
What bots see per page
NennBot reads up to 50 pages, robots.txt first, and only reads.
Failed checks first
"Fix with Nenn" starts a draft. You approve it before anything changes.
How NennBot reads a site, and how to spot it in your logs: NennBot.
Questions about AI and your website
Why can't ChatGPT see my website when Google can?
Google reads your site with Googlebot. ChatGPT search uses OAI-SearchBot. A robots.txt or firewall rule can treat the two bots in different ways. Also, Googlebot runs JavaScript, while most AI bots read only the HTML.
Can Perplexity see my website?
Run the same 4 checks with PerplexityBot. Perplexity says it uses PerplexityBot to show and link sites in its answers. Its fetch bot, Perplexity-User, works when a person asks, and Perplexity says it generally ignores robots.txt.
Does ChatGPT crawl my site on its own?
OpenAI uses OAI-SearchBot for its search and GPTBot for training. ChatGPT-User visits only when a person asks for a page. Each one can be allowed or blocked on its own.
Can ChatGPT read my website if I paste the link into a chat?
Then ChatGPT-User fetches the page. A firewall can still block it, and text drawn by JavaScript can still be missing. If the chat says it cannot open the page, start with checks 2 and 3.
Do I need an llms.txt file for AI to see my site?
No. Google says its AI features need no special files or markup. An llms.txt file does no harm, but clear text that bots may read matters more.
See what AI bots see on your website
The Website check in the trial tests your robots.txt, firewall and pages for AI crawlers. Then the weekly check shows if AI names you.
Card needed. Cancel before day 7 and nothing is charged.
Can ChatGPT read my pricing page?
- 1Your website?
Keep reading
From the bot choice to the other reasons AI names a rival.