Technical Setup

Robots.txt for AI Crawlers: Who Gets on the Guest List

Moonlit Tools teamUpdated 5 min read

Short answer

To be cited by AI assistants, let their search bots in: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Training bots like GPTBot, ClaudeBot and Google-Extended are a separate choice. Blocking them limits use of your content for training AI models, but it doesn't remove you from AI search results.

  • Search bots decide whether assistants can find and link to you. Keep them allowed.
  • Saying no to training bots is about how your content is used, not whether you show up in search. Google-Extended also affects Gemini app answers.
  • A bot follows only the group of rules that names it, so repeat shared rules in each group.
  • Your host or security service can block bots before they ever read robots.txt.
On this page
  1. Three Kinds of AI Bots
  2. Every Bot by Name
  3. Two Setups You Can Copy
  4. Spot the Problem
  5. Other Ways Sites Hide From AI Search by Accident
  6. Five-Minute Check
  7. Frequently Asked Questions
  8. Sources

Robots.txt is a small text file at yoursite.com/robots.txt. Think of it as the guest list for your website. Search engines and AI tools send bots (often called crawlers) to read your site, and well-behaved bots check this list before they come in.

It's also one of the easiest places to lock out the assistants you want citing you. One copied line can do it.

Most bots check robots.txt, and a few come with a person. Three columns of bots. Let them in: search bots OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, which are how assistants find and cite you. Your call: training bots GPTBot, ClaudeBot and Google-Extended, used for AI training, and Google-Extended also for Gemini answers. Blocking them won't hide you from search. Plus-ones: bots sent by a person, ChatGPT-User, Claude-User and Perplexity-User, which visit when asked, and some skip the list. robots.txt is your guest list Most bots check it. A few come with a person. ✓ LET THEM IN Search bots OAI-SearchBotClaude-SearchBotPerplexityBotGooglebot How assistants find and cite you ? YOUR CALL Training bots GPTBotClaudeBotGoogle-Extended AI training, and Gemini answers for Google-Extended. Search still finds you. ↪ PLUS-ONES Sent by a person ChatGPT-UserClaude-UserPerplexity-User Visit when asked. Some skip the list.

Three Kinds of AI Bots

Search bots list your pages so assistants can find and link to them. Keep these on the list.

Training bots collect content that may be used to train future AI models. Blocking them is a choice about how your content is used. It doesn't affect Google Search, ChatGPT search or Claude search results. Google-Extended also covers grounding in Gemini apps, though, so blocking it can mean Gemini draws on your pages less.

User-triggered bots visit because a person asked an assistant to look at a page. OpenAI and Perplexity both say these bots may not follow robots.txt, since a person made the request.

Every Bot by Name

The first column is the name each bot uses. It's what you type after User-agent: in robots.txt.

Bot nameCompanyJobIf you block it
OAI-SearchBotOpenAIChatGPT searchYou won't appear in ChatGPT search answers
GPTBotOpenAIModel trainingSignals your content shouldn't be used for training
ChatGPT-UserOpenAIVisits when a user asksRobots.txt may not apply, per OpenAI
Claude-SearchBotAnthropicClaude search resultsClaude can't index you for search
ClaudeBotAnthropicModel trainingFuture content is left out of training
Claude-UserAnthropicVisits when a user asksClaude can't fetch your pages for users
PerplexityBotPerplexityPerplexity search resultsLess visibility in Perplexity
Perplexity-UserPerplexityVisits when a user asksGenerally ignores robots.txt, per Perplexity
GooglebotGoogleSearch, AI Overviews, AI ModeYou disappear from Google Search
Google-ExtendedGoogleGemini training and Gemini app answersNo effect on Search or AI Overviews

Google-Extended isn't a separate bot. It's a name Google reads in robots.txt. Google crawls with its usual bots and checks this line to decide whether your content can be used for Gemini.

These lists change. Check each company's page (linked at the bottom) before a big edit.

Two Setups You Can Copy

Option A: Allow AI search, block training. A good fit if you want citations but would rather your content not be used to train AI models.

# AI search: come on in
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/

# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

The training group covers the main companies, not every AI crawler. Apple says blocking Applebot-Extended doesn't remove you from its search results.

Option B: Let everyone in. The widest reach, with private areas still blocked.

User-agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

To use one:

  1. Copy the setup you want.
  2. Change /account/ and /checkout/ to the folders on your site you want kept private, or delete those lines.
  3. Change https://www.example.com/sitemap.xml to your own sitemap's address.
  4. Replace everything in your robots.txt file with it. Your website platform, host or developer can tell you where that file is edited.
  5. Open yoursite.com/robots.txt in your browser. You should see your new text.

The rule that trips everyone up

A bot follows only the group of rules that names it most specifically. Once you add a User-agent: GPTBot group, GPTBot ignores your User-agent: * rules completely. That's why Option A repeats Disallow: /account/ in the search group.

Spot the Problem

Before: oops

User-agent: *
Disallow: /

Left over from a test version of the site. Every bot, including Googlebot, is turned away.

After: fixed

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Public pages open, private folder closed, sitemap listed.

Other Ways Sites Hide From AI Search by Accident

Five-Minute Check

  1. Open yourdomain.com/robots.txt in your browser. It should load as plain text.
  2. Read each User-agent group. Make sure the search bots you want aren't under a Disallow: / line.
  3. In Google Search Console, go to Settings. Next to robots.txt, click Open Report to see what Google last fetched.
  4. Ask your host or CDN whether any bot settings block AI bots.
  5. Give it time after a change. OpenAI says its search systems take about 24 hours to catch up.

Where Moonlit Tools Fits

Want the quick version? The free AI Crawler Checker reads any site's robots.txt and shows which of these bots it allows. A Moonlit Tools scan runs the same check and counts search crawlers blocked on most pages toward AI Readiness. Blocking training-only crawlers never lowers it.

Frequently Asked Questions

Should I block AI crawlers?

It depends on your goals. Want citations and visits from assistants? Let the search bots in. Blocking training bots is a separate choice and doesn't hide you from AI search.

Does Crawl-delay work?

For some bots. Crawl-delay is an unofficial line that asks bots to slow down. Anthropic says it supports it. Google ignores it. If bots are slowing your server, ask your host about rate limiting, which caps how often a bot can visit.

Will blocking Google-Extended remove me from AI Overviews?

No. Google says AI Overviews and AI Mode follow Googlebot's rules like the rest of Search. To leave AI Overviews and AI Mode while staying in regular results, use the Search generative AI setting in Search Console (Settings > Search generative AI). Snippet tags like nosnippet, data-nosnippet and max-snippet also limit what Search shows from a page.

Sources

See What Google and AI Tools Can Read

Moonlit Tools reads your site the way search engines and AI assistants do, then tells you what to fix first and shows you why. Scan your own website free. No card needed.

Scan Your Site Free Or apply for 90 days of Launch free