Where AI answers get their sources: Google, ChatGPT and Perplexity

Where AI answers get their sources: Google, ChatGPT and Perplexity

Written by

in

The verdict

AI answers do not come from a new kind of index. Google’s AI features use Google’s search index. ChatGPT search and Perplexity run their own search crawlers, and blocking them keeps you out of their answers. So the job is the old one: be crawlable by the right bots, be indexed, and be the best page on the subject.

Business owners are now being sold “AI visibility” as a new discipline with new tools and new fees. Before buying any of it, it helps to know where the main AI answer engines actually get the pages they cite. Each one publishes enough to answer that question, and the answer is less mysterious than the sales pitch suggests.

Google: AI Overviews and AI Mode

Google’s page on AI features and your website is unusually direct. To be shown as a supporting link, “a page must be indexed and eligible to be shown in Google Search with a snippet.” It also says “there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary,” and that there is “no special schema.org structured data that you need to add.”

Both features may use what Google calls “query fan-out”: running several related searches across subtopics and combining the results. That is still search, run several times. I explain what it means for content planning in query fan-out is how Google searches, not a list of pages to write, and why any proposal that ignores this should be read carefully in AI Overviews run on the index you already have.

Google reports AI feature appearances inside the normal Performance report in Search Console, under the Web search type, and its Generative AI report breaks them out, which I go through in the AI report in Search Console. There is still no separate system to optimise for.

ChatGPT search

OpenAI separates its crawlers by purpose in its documentation on OpenAI crawlers. OAI-SearchBot “is for search,” and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” GPTBot is different: it crawls content “that may be used in training” OpenAI’s models, and blocking it does not affect search. A third agent, ChatGPT-User, fetches pages when a user asks ChatGPT something, and OpenAI notes that for these user-initiated visits “robots.txt rules may not apply.”

So a site can decline training and still appear in ChatGPT search, by blocking GPTBot and allowing OAI-SearchBot. It is easy to block all of them by accident, through a security plugin or a robots.txt copied from somewhere else.

Perplexity

Perplexity’s crawler documentation describes PerplexityBot as “designed to surface and link websites in search results on Perplexity,” says it “is not used to crawl content for AI foundation models,” and recommends allowing it. Its second agent, Perplexity-User, fetches pages when a user asks a question and “generally ignores robots.txt rules.”

Microsoft Copilot

Copilot is built on Bing’s search. For a business site that makes Bing Webmaster Tools worth the ten minutes it takes to set up, and Bing Places worth claiming, because that is the data Copilot has to work with. The rest of the non-Google picture is in the other 15% of search.

What this means for a business site

  • Be indexed in Google and Bing. Most AI answers rest on one or the other, or on a crawler that behaves like one.
  • Check robots.txt for the search bots. OAI-SearchBot and PerplexityBot, at least. Blocking training crawlers is a separate decision.
  • Keep the answer in the page text. Not in images, not behind a click, not loaded only by a script that may not run.
  • Be the best page on the subject. AI answers cite pages for the same reasons search ranks them.
  • Measure in Search Console and analytics. Not with a third-party “AI visibility” score.

For local, trades and ecommerce businesses I do not recommend blocking AI search crawlers. Being cited in an answer is still being found. Training crawlers are a judgement call, and I explain the Google side of it in blocking Google-Extended will not take you out of AI Overviews.

The short version

  1. Google’s AI features use Google’s index. Indexed and snippet-eligible is the bar.
  2. ChatGPT search has its own crawler. Block OAI-SearchBot and you are not shown.
  3. Training and search are separate bots. You can allow one and block the other.
  4. Perplexity also crawls for search. Its search bot is not used for training.
  5. Copilot is built on Bing. Set up Bing Webmaster Tools.
  6. The job is still SEO. Crawlable, indexed, and the best answer.

If a site should be ranking and it isn’t, that’s the work I do. Technical SEO includes checking which search and AI crawlers can actually reach your pages.

Headshot of Mehul Dedhia, SEO consultant

Written by Mehul Dedhia

I’ve been doing SEO for about fifteen years. Most of my work isn’t building new websites. It’s figuring out why sites that should rank don’t. Local businesses, ecommerce and Shopify brands, affiliate sites, small brands and enterprise. If a site is stuck or has lost rankings, that’s usually where I come in.

Connect with me on LinkedIn