NennBot, the crawler of Nenn

NennBot reads a website when someone asks Nenn to check it. It does not crawl the web on its own.

Updated

What NennBot is

Nenn is an AI visibility tool. It shows businesses how ChatGPT, Perplexity and Google AI talk about them.

To give advice, it reads the business's own website. NennBot is the part of Nenn that does this reading.

When it visits

  • A Nenn customer adds their website. The first check starts at once.
  • A customer starts a new check. Each site gets at most 1 new check per hour.
  • Someone types the site into the test run on nenn.io/check.
  • A customer publishes a page with Nenn. It then reads the logo, colour and font of their site.
  • Nenn shows a site's icon next to its name in the competitor list.

Icon reads, style reads and the test run are a few single requests. They do not check robots.txt and are not spaced one per second. The style read may also load your stylesheets from another server, like a CDN.

What it reads, in order

A full check always follows the same path. It stops early when robots.txt says no.

  1. robots.txt

    Read first. Blocked pages are never opened.

    • robots.txt blocks NennBotStop. Nenn tells its customer.

    • robots.txt does not answerHome page only.

  2. Home page

    /

    The start of every check.

  3. llms.txt

    A short guide for AI tools, if the site has one.

  4. Sitemap

    /sitemap.xml

    From robots.txt or /sitemap.xml. An index opens up to 3 more sitemaps.

  5. Pages

    AI-cited pages first, then sitemap pages, then internal links. Up to 50 pages in the smallest check.

  • GET only
  • 1 request per second
  • Same site only
How one check visits your site

How it behaves in a full check

GET only
It only reads. It never sends forms and never logs in.
1 per second
One request per second per site. www and the bare domain share this limit.
8 seconds
It stops waiting for a page after 8 seconds.
3 MB
It reads at most 3 MB of one page.
5 redirects
Each new address is checked again.
Same site
It never follows links to other websites.
24 hours
A page it read in the last 24 hours is not loaded again.
15 minutes
One check stops reading new pages after 15 minutes.
50 pages
The smallest limit. Bigger plans read more, never over 5,000 in one check.
Public only
It only connects to public addresses, never to private networks.

One more visit: a full check loads your home page once as each AI crawler it tests. Examples are GPTBot and PerplexityBot. This shows if your server lets them in. These requests use that crawler's user agent, not NennBot.

Block or allow NennBot in robots.txt

In a full check, NennBot reads robots.txt first. It never opens a page that robots.txt blocks for it.

  • Block NennBot from full checks
    User-agent: NennBot
Disallow: /
  • Block one folder
    User-agent: NennBot
Disallow: /private/
  • Allow NennBot while other bots are blocked
    User-agent: *
Disallow: /

User-agent: NennBot
Allow: /
  • A group for NennBot replaces the * group for it. Rules in * then do not apply to NennBot.
  • It reads Allow and Disallow, and the * and $ wildcards. The longest matching rule wins. On a tie, Allow wins.
  • If your robots.txt blocks the home page, Nenn stops and tells its customer.

The test run on nenn.io/check

This check reads much less. It loads the home page, the site icon, robots.txt and llms.txt.

On German sites without a city on the home page, it may also open /impressum or /kontakt.

It uses the same user agent. It reads up to 2 MB per page and keeps the result for 24 hours.

It does not check robots.txt before it opens these pages.

Contact

Seen NennBot and have a question? Write to hello@nenn.io.

New to Nenn? See what Nenn does.