Free Tool
AI Crawler Checker
Enter a domain. The checker reads its robots.txt and shows which AI search, user and training crawlers may read the homepage. It uses the same rules Moonlit Tools uses in every scan.
Results
| Crawler | Homepage | Job | Matching rule | Company |
|---|
How the Check Works
- It fetches
https://your-domain/robots.txtonce, follows up to 3 redirects and reads up to 256 KB. Nothing is saved. - Each crawler follows the group that names it most specifically, or the
*group when none does. The checker applies the longest matching Allow or Disallow rule for the homepage path,/. - No robots.txt (a 404) means crawlers that follow robots.txt may read every page. An error, a timeout or an HTML page in place of the file shows as Unknown, never as Allowed.
Crawlers It Checks
| Crawler | Company | Job |
|---|---|---|
Googlebot | Search and AI search | |
Bingbot | Microsoft | Search and AI search |
OAI-SearchBot | OpenAI | Search and AI search |
Claude-SearchBot | Anthropic | Search and AI search |
PerplexityBot | Perplexity | Search and AI search |
ChatGPT-User | OpenAI | Fetches when a person asks |
Perplexity-User | Perplexity | Fetches when a person asks |
Claude-User | Anthropic | Fetches when a person asks |
GPTBot | OpenAI | Model training |
ClaudeBot | Anthropic | Model training |
Google-Extended | Model training | |
Applebot-Extended | Apple | Model training |
CCBot | Common Crawl | Model training |
Bytespider | ByteDance | Model training |
Meta-ExternalAgent | Meta | Model training |
Amazonbot | Amazon | Model training |
Blocking a training crawler is a choice about how your content is used. Blocking a search crawler can keep you out of that assistant's answers. The guide Robots.txt for AI Crawlers explains each one and has setups you can copy. Moonlit Tools' own crawler is described on About MoonlitBot.
Check More Than robots.txt
A Moonlit Tools scan also checks snippet controls, llms.txt, indexing signals and whether your pages have text AI tools can read, then scores AI Readiness. Scan your website free.
Scan Your Site Free Or explore the demo