Moonlit

How-To · Technical Setup

Robots.txt for AI Crawlers: Who Gets on the Guest List

Which AI crawlers to let in, which ones are optional, and copy-ready robots.txt examples, plus the sneaky mistakes that quietly hide sites from AI search.

By the Moonlit teamPublished 4 min read

Short answer

To be cited by AI assistants, let their search crawlers in, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Training crawlers like GPTBot, ClaudeBot and Google-Extended are a separate choice. Blocking them limits use of your content for model training but doesn't remove you from AI search results.

Key Takeaways

  • Let search crawlers in: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot.
  • Training crawlers like GPTBot, ClaudeBot and Google-Extended are optional and don't affect AI search visibility.
  • A crawler follows only the most specific group that names it, so repeat shared rules in each group.
  • Check your CDN and host bot settings too. They can block crawlers before robots.txt is read.

Robots.txt is a tiny text file at yoursite.com/robots.txt. It's basically the guest list for your website. Well-behaved crawlers read it at the door before they wander in.

It's also one of the easiest places to accidentally lock out the very assistants you want citing you. One copied line can do it.

Robots.txt as a guest list for AI crawlers Three columns of crawlers. Search crawlers, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, are on the list and are how assistants find and cite you. Training crawlers, GPTBot, ClaudeBot and Google-Extended, are your call and don't affect AI search visibility. User-triggered agents, ChatGPT-User, Claude-User and Perplexity-User, arrive because a person asked, and some of them do not check the list. robots.txt is your guest list Most crawlers check it at the door. A few walk in with a guest. ✓ LET THEM IN Search crawlers OAI-SearchBotClaude-SearchBotPerplexityBotGooglebot How assistants find and cite you ? YOUR CALL Training crawlers GPTBotClaudeBotGoogle-Extended Model training only. Blocking won't hide you. ↪ PLUS-ONES User-triggered agents ChatGPT-UserClaude-UserPerplexity-User Arrive with a person. Some skip the list.
Three kinds of AI visitors. Only one of them decides whether assistants can find and cite you.

The Short Version of Who's Who

Search crawlers index your pages so assistants can find and link to them. Keep these on the list.

Training crawlers collect content that may be used to train future models. Blocking them is a content-use decision, not a visibility decision.

User-triggered agents show up because a person asked an assistant to look at a page. OpenAI and Perplexity both say their user-triggered agents may not follow robots.txt, since a human made the request.

The Full Cast

User agentCompanyJobIf you block it
OAI-SearchBotOpenAIChatGPT searchYou won't appear in ChatGPT search answers
GPTBotOpenAIModel trainingSignals your content shouldn't be used for training
ChatGPT-UserOpenAIVisits when a user asksRobots.txt may not apply, per OpenAI
Claude-SearchBotAnthropicClaude search resultsClaude can't index you for search
ClaudeBotAnthropicModel trainingFuture content is left out of training
Claude-UserAnthropicVisits when a user asksClaude can't fetch your pages for users
PerplexityBotPerplexityPerplexity search resultsLess visibility in Perplexity
Perplexity-UserPerplexityVisits when a user asksGenerally ignores robots.txt, per Perplexity
GooglebotGoogleSearch, AI Overviews, AI ModeYou disappear from Google Search
Google-ExtendedGoogleOther Google AI uses, like trainingNo effect on Search or AI Overviews

These lists change, so peek at each company's docs (linked at the bottom) before a big edit.

Two Copy-Ready Setups

Option A: Welcome AI search, skip training. Popular with businesses that want citations but would rather not feed model training.

# AI search: come on in
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/

# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Option B: Open house. Maximum reach, private areas still private.

User-agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

Spot the Problem

Before: oops

User-agent: *
Disallow: /

Left over from a staging site. Every crawler, including Googlebot, is turned away.

After: fixed

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Public pages open, private folder closed, sitemap listed.

Five-Minute Check

  1. Open yourdomain.com/robots.txt. It should load as plain text.
  2. Read each User-agent group. Make sure the search crawlers you want aren't disallowed.
  3. Open the robots.txt report in Google Search Console to see what Google last fetched.
  4. Ask your host or CDN whether any bot rules block AI crawlers.
  5. Be patient after changes. OpenAI says its search systems take about 24 hours to adjust.

Frequently Asked Questions

Should I block AI crawlers?

Depends on your goals. Want citations and visits from assistants? Let the search crawlers in. Blocking training crawlers is a separate decision and doesn't hide you from AI search.

Does Crawl-delay work?

For some. Anthropic says it supports the non-standard Crawl-delay extension. Google ignores it. If crawling is straining your server, rate limiting at your host is more reliable.

Will blocking Google-Extended remove me from AI Overviews?

No. Google says AI Overviews and AI Mode follow Googlebot's rules like the rest of Search. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet or noindex.

Sources