About MoonlitBot

MoonlitBot is the crawler behind Moonlit website scans. If it shows up in your server logs, someone with a Moonlit account asked for a scan of your website or turned on weekly rescans for it.

How to Recognize It

MoonlitBot sends this user agent. The version number can change.

MoonlitBot/0.1 (+https://moonlit.tools/crawler; website-intelligence scan)

What It Requests

• Public pages, your robots.txt and your sitemap, using GET and HEAD requests only. It does not sign in or fill in forms.
• Pages on the website being scanned. It does not follow links to other websites.
• For automated accessibility checks it loads pages in a headless browser with the same user agent. That browser blocks requests to other domains and skips images, video and fonts.

How to Limit or Block It

MoonlitBot follows robots.txt. It uses a group named MoonlitBot when you have one, and your * group otherwise. It pauses between requests, and waits longer when your robots.txt sets a Crawl-delay.

User-agent: MoonlitBot
Disallow: /private/
Crawl-delay: 2

To block it entirely, use Disallow: / in its group. Pages you block are left out of the scan and are not counted against the website's coverage.

Questions

Email hello@moonlit.tools if MoonlitBot is causing a problem on your website. Include the time of the requests and the domain.