Allowing AI crawlers
To let AI search engines read your pages, allow their crawlers in robots.txt and make sure no firewall or bot-protection rule blocks them at the network edge. Allow GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), Bingbot (Copilot) and Google-Extended (Google's generative products), and keep Googlebot allowed for AI Overviews. If a bot cannot reach a page, no on-page work can make that page citable.
- →Allow GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot in robots.txt, and keep Googlebot allowed.
- →robots.txt is a request, not a lock: a firewall or WAF can still block an allowed bot, so check both.
- →List your XML sitemap in robots.txt so crawlers can find every page.
- →This is a template fix. One file governs crawler access for the whole site.
Why crawler access is a prerequisite
An AI engine can only quote a page it has read. Answer-first openers, schema and sourced evidence do nothing if the crawler that feeds the engine is turned away at the door. Crawler access is the Findable check that gates all the others, which is why "robots allows AI bots" and "AI crawlers not network-blocked" are both scored, and why access fixes sit in Phase 01 Foundation of the Action Plan.
This is a template fix. A single robots.txt file governs access for every page on the site, so allowing the crawlers is one change that unblocks everything. See template-fixes-vs-page-fixes for why site-wide fixes move your score first.
The robots.txt rules to allow AI crawlers
Each engine sends a named user-agent. To allow a bot, give it an empty Disallow: line, which permits everything. Add a Sitemap: line so crawlers can find every URL.
# robots.txt — allow AI search crawlers to read the site
User-agent: GPTBot
Disallow:
User-agent: OAI-SearchBot
Disallow:
User-agent: ClaudeBot
Disallow:
User-agent: PerplexityBot
Disallow:
User-agent: Google-Extended
Disallow:
User-agent: Bingbot
Disallow:
User-agent: Googlebot
Disallow:
Sitemap: https://example.com/sitemap.xmlWhat each user-agent is:
| User-agent | Belongs to | Powers |
|---|---|---|
| GPTBot, OAI-SearchBot | OpenAI | ChatGPT and ChatGPT search |
| ClaudeBot | Anthropic | Claude |
| PerplexityBot | Perplexity | Perplexity answers |
| Bingbot | Microsoft | Microsoft Copilot (via the Bing index) |
| Google-Extended | Whether Google may use your content in Gemini and AI Overviews grounding | |
| Googlebot | Google Search and the index behind AI Overviews |
Google-Extended is a control token, not a crawler. It governs generative use of your content and does not affect ordinary crawling or Search ranking. Allow it to let Google use your pages in AI answers.
The block to check for
The most common cause of an invisible site is an accidental blanket disallow. A single global block stops every crawler, AI or otherwise.
# This blocks everything. Remove it if you want to be crawled.
User-agent: *
Disallow: /A per-bot disallow does the same to one engine, and many stacks and templates ship with AI bots blocked by default:
User-agent: GPTBot
Disallow: /If you find either pattern, that is why the bot cannot read the site. Replace the Disallow: / with an empty Disallow: for the bots you want to allow.
User-agent: * group for that bot. If you allow GPTBot explicitly but also have User-agent: * / Disallow: /, GPTBot follows its own group and is allowed. Set each AI bot's own group to be sure.robots.txt is a request, not a lock
robots.txt only asks well-behaved crawlers to stay out. It does not physically stop a request. A firewall, WAF or bot-protection service (for example an aggressive bot-fight mode, rate limiting, or an IP or user-agent block) is what actually blocks a crawler, and it can stop a bot you have allowed in robots.txt.
So if a crawl is blocked even though robots.txt looks correct, check the edge:
See my-crawl-was-blocked for diagnosing a specific block and agent-ready-and-crawler-access for how reachability, server rendering and speed together decide whether an engine can retrieve your pages.
List your sitemap
An XML sitemap helps crawlers discover every page, especially ones that are not well linked internally. Reference it in robots.txt with a Sitemap: line, as shown above, and make sure the sitemap itself is reachable and lists your canonical URLs. The sitemap is not scored as a page, but its presence is a Findable signal.