Technical Setup
Robots.txt for AI Crawlers: Who Gets on the Guest List
To be cited by AI assistants, let their search bots in: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Training bots like GPTBot, ClaudeBot and Google-Extended are a separate choice. Blocking them limits use of your content for training AI models, but it doesn't remove you from AI search results.
- Search bots decide whether assistants can find and link to you. Keep them allowed.
- Saying no to training bots is about how your content is used, not whether you show up in search. Google-Extended also affects Gemini app answers.
- A bot follows only the group of rules that names it, so repeat shared rules in each group.
- Your host or security service can block bots before they ever read robots.txt.
On this page
Robots.txt is a small text file at yoursite.com/robots.txt. Think of it as the guest list for your website. Search engines and AI tools send bots (often called crawlers) to read your site, and well-behaved bots check this list before they come in.
It's also one of the easiest places to lock out the assistants you want citing you. One copied line can do it.
Three Kinds of AI Bots
Search bots list your pages so assistants can find and link to them. Keep these on the list.
Training bots collect content that may be used to train future AI models. Blocking them is a choice about how your content is used. It doesn't affect Google Search, ChatGPT search or Claude search results. Google-Extended also covers grounding in Gemini apps, though, so blocking it can mean Gemini draws on your pages less.
User-triggered bots visit because a person asked an assistant to look at a page. OpenAI and Perplexity both say these bots may not follow robots.txt, since a person made the request.
Every Bot by Name
The first column is the name each bot uses. It's what you type after User-agent: in robots.txt.
| Bot name | Company | Job | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search | You won't appear in ChatGPT search answers |
GPTBot | OpenAI | Model training | Signals your content shouldn't be used for training |
ChatGPT-User | OpenAI | Visits when a user asks | Robots.txt may not apply, per OpenAI |
Claude-SearchBot | Anthropic | Claude search results | Claude can't index you for search |
ClaudeBot | Anthropic | Model training | Future content is left out of training |
Claude-User | Anthropic | Visits when a user asks | Claude can't fetch your pages for users |
PerplexityBot | Perplexity | Perplexity search results | Less visibility in Perplexity |
Perplexity-User | Perplexity | Visits when a user asks | Generally ignores robots.txt, per Perplexity |
Googlebot | Search, AI Overviews, AI Mode | You disappear from Google Search | |
Google-Extended | Gemini training and Gemini app answers | No effect on Search or AI Overviews |
Google-Extended isn't a separate bot. It's a name Google reads in robots.txt. Google crawls with its usual bots and checks this line to decide whether your content can be used for Gemini.
These lists change. Check each company's page (linked at the bottom) before a big edit.
Two Setups You Can Copy
Option A: Allow AI search, block training. A good fit if you want citations but would rather your content not be used to train AI models.
# AI search: come on in
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
# Model training: no thanks
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Everyone else
User-agent: *
Allow: /
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
The training group covers the main companies, not every AI crawler. Apple says blocking Applebot-Extended doesn't remove you from its search results.
Option B: Let everyone in. The widest reach, with private areas still blocked.
User-agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
To use one:
- Copy the setup you want.
- Change
/account/and/checkout/to the folders on your site you want kept private, or delete those lines. - Change
https://www.example.com/sitemap.xmlto your own sitemap's address. - Replace everything in your robots.txt file with it. Your website platform, host or developer can tell you where that file is edited.
- Open
yoursite.com/robots.txtin your browser. You should see your new text.
The rule that trips everyone up
A bot follows only the group of rules that names it most specifically. Once you add a User-agent: GPTBot group, GPTBot ignores your User-agent: * rules completely. That's why Option A repeats Disallow: /account/ in the search group.
Spot the Problem
Before: oops
User-agent: *
Disallow: /
Left over from a test version of the site. Every bot, including Googlebot, is turned away.
After: fixed
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Public pages open, private folder closed, sitemap listed.
Other Ways Sites Hide From AI Search by Accident
- "Block all AI" lists from the internet. These often include search bots like OAI-SearchBot and PerplexityBot, not just training bots.
- Bot protection at your host or CDN (a service that sits in front of your site to speed it up and filter traffic). These settings can block bots before they read robots.txt. Google's AI features guidance says to check your CDN and hosting.
- Blocking your CSS and JavaScript folders. Googlebot needs those files to see your pages the way people do.
- Using robots.txt to hide pages. It controls which pages bots visit, not which pages appear in search. A blocked page can still show up if other sites link to it. To keep a page out of search, add a
noindextag to the page and leave it unblocked. - No sitemap line. Adding
Sitemap:with your full sitemap address helps every bot find your pages faster.
Five-Minute Check
- Open
yourdomain.com/robots.txtin your browser. It should load as plain text. - Read each
User-agentgroup. Make sure the search bots you want aren't under aDisallow: /line. - In Google Search Console, go to Settings. Next to robots.txt, click Open Report to see what Google last fetched.
- Ask your host or CDN whether any bot settings block AI bots.
- Give it time after a change. OpenAI says its search systems take about 24 hours to catch up.
Where Moonlit Tools Fits
Want the quick version? The free AI Crawler Checker reads any site's robots.txt and shows which of these bots it allows. A Moonlit Tools scan runs the same check and counts search crawlers blocked on most pages toward AI Readiness. Blocking training-only crawlers never lowers it.
Frequently Asked Questions
Should I block AI crawlers?
It depends on your goals. Want citations and visits from assistants? Let the search bots in. Blocking training bots is a separate choice and doesn't hide you from AI search.
Does Crawl-delay work?
For some bots. Crawl-delay is an unofficial line that asks bots to slow down. Anthropic says it supports it. Google ignores it. If bots are slowing your server, ask your host about rate limiting, which caps how often a bot can visit.
Will blocking Google-Extended remove me from AI Overviews?
No. Google says AI Overviews and AI Mode follow Googlebot's rules like the rest of Search. To leave AI Overviews and AI Mode while staying in regular results, use the Search generative AI setting in Search Console (Settings > Search generative AI). Snippet tags like nosnippet, data-nosnippet and max-snippet also limit what Search shows from a page.
Sources
- Overview of OpenAI crawlers, OpenAI
- Anthropic's crawlers, Anthropic
- Perplexity crawlers, Perplexity
- AI features and your website, Google Search Central
- Introduction to robots.txt, Google Search Central
- Google's common crawlers (Google-Extended), Google
- About Applebot, Apple Support
- Search generative AI control, Search Console Help
See What Google and AI Tools Can Read
Moonlit Tools reads your site the way search engines and AI assistants do, then tells you what to fix first and shows you why. Scan your own website free. No card needed.
Scan Your Site Free Or apply for 90 days of Launch free