Technical access for AI crawlers

An AI engine can only cite a page it can reach and read. Technical access is the set of controls that decide whether it can, and those controls are set engine by engine.

A site that ranks well in Google can still be missing from ChatGPT, Perplexity, or Copilot answers for reasons that have nothing to do with content. Each engine fetches pages with its own crawler or draws on its own index, and each one answers to a different setting.

Which crawler and which control serve each engine?

An AI crawler is a program an AI company runs to fetch pages. Most AI companies run more than one, for different jobs: building a search index, fetching a page a user asked about, and collecting training data. The controls are independent, so a site can turn one off and keep the others.

The map below is drawn from each company's own documentation.

Sources: OpenAI's crawler overview, Anthropic's crawler help page, Perplexity's crawler documentation, Google's pages on common crawlers and AI features, and the Bing Webmaster Blog announcement of the Bing Chat controls, written before Bing Chat was renamed Copilot.

Two rows in the map catch experienced SEO teams out. Google states that Google-Extended has no effect on Google Search, so blocking it does not remove a page from AI Overviews or AI Mode. Those two surfaces follow the ordinary Search controls, and the snippet controls are what limit how much of a page they can show.

Copilot is the other. It draws on the Bing index, so a site that has never checked how Bing crawls it has never checked Copilot. Under Bing's controls, NOARCHIVE keeps a page out of the chat answers, and NOCACHE limits what the answer can use to the URL, title, and snippet.

Some engines publish no crawler documentation at all. For those, access cannot be confirmed from the site's side. The honest report is that access to that engine is unknown.

Which layer is blocking, and in what order should it be checked?

Crawlability is lost at four layers, and they are worth checking from the outside in. An outer block makes every setting further in irrelevant, so starting inside wastes the work.

1. Firewall and content delivery network (CDN) rules. These run before the site sees the request, and they can refuse a crawler that robots.txt allows. Since July 2025, Cloudflare has blocked AI crawlers by default on new domains and asks the owner to choose. A site on Cloudflare, or behind any bot-management service, needs that setting read first.
2. robots.txt. robots.txt is a request that the documented crawlers say they follow, written separately for each crawler name. A rule meant for a training crawler can be written so broadly that it also catches the search crawler from the same company.
3. What the server sends. JavaScript rendering builds content in the browser after the page loads. Google documents that Googlebot renders JavaScript, and none of the AI companies' crawler pages above says the same, so the safe assumption is that a crawler sees only the HTML the server returns. Content that matters belongs in that HTML.
4. Index and snippet controls. Indexability decides whether a fetched page can be stored and shown. A noindex tag, a nosnippet tag, or Bing's NOARCHIVE removes a page from answers even when every crawler can reach it.

Server logs show which layer is at work. A crawler that requests robots.txt and then none of the pages is following a robots.txt rule, at layer two. One that fetches pages successfully points further in, and one that never appears is either stopped at layer one or has not visited yet, which the firewall settings decide between.

How does measurement point at an access problem?

Access problems rarely announce themselves. They show up in the measurement described in chapter 3, as a gap that follows an engine rather than a topic.

Two patterns are worth reading for:

• A page cited on one engine and never on another. The content is evidently usable, so the difference sits with the second engine. Its row in the map above says which crawler and which control to check.
• A brand mentioned often whose own pages are cited nowhere. Engines know the brand from other sites but are not using the brand's pages. A site-wide layer, usually the firewall or robots.txt, is the first suspect.

Access is checked before content is rewritten. Rewriting a page that no crawler can read changes nothing in the answers, and the measurement would then wrongly suggest the new content failed.

Do sitemaps and llms.txt help?

A Sitemap lists a site's pages so crawlers can find them. Google and Bing both use sitemaps for discovery, and since AI Overviews, AI Mode, and Copilot draw on those two indexes, a current sitemap helps those surfaces. It grants no access that a layer above has removed.

llms.txt is a proposed plain-text file that points language models to a site's most useful pages. OpenAI and Perplexity publish one for their own documentation, but neither company's crawler page says its crawlers read one on other sites.

It is cheap to publish and does no harm. Google states in its guide to AI features and websites that no AI text files are needed to appear in AI Overviews or AI Mode, and llms.txt does not replace any of the four layers.

Structured data sits closer to content than to access: it describes what a page is about once a crawler can read it. Chapter 5 covers it with the rest of the content an engine can use, under structured data.

What does technical access not decide?

Access makes a page eligible to be used, and nothing more. Once the layers are clear, the question becomes what makes a page worth citing, which the next chapter on content takes up.

Zumi reports citations per engine, which shows where a brand's pages are used as sources and where they never appear. Book a demo