Moonlit

Deep Dive · How It Works

How AI Assistants Find and Cite Websites (A Field Guide)

The long version. Follow a question from someone's phone to an AI answer with your link in it, see where pages fall out along the way, and learn what to fix at each step.

By the Moonlit teamPublished Updated 7 min read

Short answer

For most specific or current questions, AI assistants search an index of web pages, read the best matches and cite the pages they used. Your page has to survive five steps to be cited, crawled, indexed, retrieved, read and chosen, and each step has its own common failure, from a blocked crawler to a page that never actually answers the question.

Key Takeaways

  • Citations come from live retrieval, not from training data.
  • A page has to be crawled, indexed, retrieved, read and chosen before it gets cited.
  • Many AI crawlers don't run JavaScript, so text has to be in the HTML.
  • Pages that answer the question in the first sentences are the easiest to cite.
  • Troubleshoot in order: access, indexing, readability, matching, then answer quality.

Picture someone standing in their yard in October, phone in hand, asking an assistant: "When should I aerate my lawn?" A few seconds later they get a tidy two-sentence answer with little numbered links underneath.

Behind that answer is a surprisingly physical process. Bots fetched pages. Indexes stored them. Something chose a handful of candidates and read them. If you've ever wondered why a competitor gets that little link and you don't, the reason is almost always hiding in one of those steps.

This guide walks the whole path.

First, Two Kinds of "Knowing"

AI assistants know about websites in two very different ways, and it helps to keep them apart.

TrainingLive retrieval
What happensPages collected in the past help train the modelPages are looked up when someone asks a question
What it affectsThe model's general background knowledgeThe specific answer and its sources
Do you get a link?NoUsually yes, as a citation
Example crawlersGPTBot, ClaudeBotOAI-SearchBot, Claude-SearchBot, PerplexityBot

Training is like reading a lot of books years ago. Retrieval is like looking something up right now with the book open on the table. Citations come from retrieval, so that's where AEO effort pays off.

The Five Steps From Page to Citation

From web page to AI citation, and where pages drop out Five stages left to right: crawled, indexed, retrieved, read, cited. Under each stage is the most common reason a page falls out: blocked by robots.txt or a firewall, missing from the sitemap or index, not relevant to the query, content hidden behind JavaScript, and no clear answer to quote. 1. Crawled A bot fetches your page 2. Indexed It's filed in a search index 3. Retrieved It matches a question 4. Read The assistant reads the text 5. Cited ✦ You're linked as a source WHERE PAGES FALL OUT Blocked byrobots.txtor a firewall Not in thesitemap, nolinks to it Doesn't matchhow peopleask Text hiddenbehindJavaScript No clearanswer toquote
Every citation survives five steps. Every step has a favorite way to lose you.

Let's walk through each one, including the most common way pages trip.

Step 1: Crawled

Before anyone asks anything, crawlers visit your site and download pages. Each assistant has its own search crawler:

Where pages fall out: a robots.txt rule, a CDN "block AI bots" toggle or a firewall that turns the crawler away before it reads a word. Google's own guidance for AI features calls out checking your CDN and hosting, not just robots.txt. Our robots.txt guide has a full list of who to let in.

Step 2: Indexed

After a page is fetched, it gets processed and stored in a search index so it can be found later. For Google, that's the same index that powers regular results, and a page has to be indexed and eligible for a snippet to show up as a link in AI Overviews.

Where pages fall out: pages that nothing links to, pages missing from the sitemap, duplicates, or pages accidentally marked noindex. A good test for Google is the URL Inspection tool in Search Console. It tells you plainly whether a page is indexed.

Step 3: Retrieved

Now someone asks a question. The assistant turns it into one or more search queries, which Bing calls grounding queries, and pulls back the most relevant candidates.

This is where wording starts to matter. If your customers ask "when should I aerate" and your page only ever says "core cultivation services," you've made the matching job harder than it needs to be.

Where pages fall out: the page is about the right topic but never uses the words people actually ask with. Headings that mirror real questions help a lot here.

Step 4: Read

The assistant reads the text of the candidate pages. Sometimes it uses what's in the index; sometimes a user-triggered agent like ChatGPT-User, Claude-User or Perplexity-User fetches the page fresh.

Here's the twist that catches a lot of modern sites. In a December 2024 study of crawler traffic, Vercel found that none of the major AI crawlers it measured executed JavaScript, including OpenAI's crawlers, ClaudeBot and PerplexityBot. Gemini, riding on Googlebot's infrastructure, could.

Where pages fall out: pages built with JavaScript that arrive as an empty shell. To a crawler that skips scripts, there's nothing there to read. Here's the two-minute test to see if that's you.

Step 5: Cited

Finally, the assistant writes its answer and links the sources it leaned on. Nobody outside these companies knows the exact selection recipe, and it changes. But the pages that get cited tend to have one thing in common: a passage that answers the question clearly enough to repeat.

Illustrative AI assistant answer with cited sources A mock chat window. The user asks when to aerate a lawn. The assistant answers in two sentences with small numbered citation markers, and below the answer three source cards are listed. The first source card, from a fictional lawn care company, is highlighted because its page answered the question in its opening line. ILLUSTRATIVE EXAMPLE When should I aerate my lawn? For cool-season grass, aerate in early fall while it's actively growing. [1] Warm-season grass does better in late spring. [2] SOURCES 1 · examplelawn.com When Should I Aerate My Lawn? Answer in the first line ✓ 2 · extension.example.edu Lawn Aeration Basics Clear, specific, dated ✕ not cited “Quality Lawn Care Since 2009!” Never answers the question
An illustrative example. The cited pages answer the question directly. The page that talks about itself doesn't make the cut.

Where pages fall out: the page talks around the topic, about the company's history, values or "quality service," without ever stating the answer.

What Makes a Page Easy to Cite

Published guidance from Google, plus a lot of common sense, points the same way.

Hard to use

  • Answer buried in paragraph four
  • "Contact us for pricing"
  • Key info only inside an image or PDF
  • Headings like "Our Approach"
  • No date, no sources

Easy to use

  • Answer in the first two sentences
  • Real price ranges and timeframes
  • Important facts written as text
  • Headings like "How Much Does It Cost?"
  • A visible updated date and linked sources

A few of these come straight from Google's AI features guidance: make content easy to find through internal links, keep important content in text form, support it with good images and video, and keep structured data consistent with what's visible on the page.

The rest is just how people like to read, too. Answer-first writing is covered in depth in How to Write Answer-First Content.

The User-Triggered Wrinkle

Not every visit from an assistant is a crawler working on its own schedule. When a person pastes your URL into ChatGPT, Claude or Perplexity and asks about it, a user-triggered agent fetches the page on that person's behalf.

These play by slightly different rules:

The practical upshot is the same as everywhere else: if the page's HTML is readable and its answer is clear, these visits go well.

A Troubleshooting Order That Saves Time

When you're not being cited, check in this order. Each step depends on the one before it.

  1. Access. Can OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot fetch the page? Check robots.txt and any bot protection at your host or CDN.
  2. Indexing. Is the page in your sitemap, linked from other pages and showing as indexed in Search Console?
  3. Readability. Is the main text in View Page Source, or only after JavaScript runs?
  4. Matching. Do your headings and opening lines use the words customers actually use?
  5. Answer quality. Does the page state a clear, specific, current answer early on?
  6. Competition. If everything above is fine, look at who is being cited. Their page might simply be more specific. That's fixable, too.

How to Watch It Happen

Frequently Asked Questions

No. GPTBot is OpenAI's crawler for content that may be used in model training. ChatGPT search uses OAI-SearchBot. You can block one and allow the other.

Does Google have a special crawler for AI Overviews?

No. Google says its AI features use regular Search crawling and indexing through Googlebot. Google-Extended is a separate control for how content is used in other Google AI systems.

Why does an assistant cite a page that's worse than mine?

Sometimes the other page is easier to read or more specific, and sometimes your page stumbled on an earlier step, like a blocked crawler or JavaScript-only text. Work through the troubleshooting order above before rewriting anything.

How quickly do changes show up?

It depends on the crawler. OpenAI says robots.txt changes for its search systems can take about 24 hours. Google notes recrawling can take days to months. Content changes usually take longer to show up than access fixes.

Sources