Agent-ready and crawler access
Crawler access is whether an AI search engine can reach your page, read it without running JavaScript, and fetch it fast enough to use. It is scored under the Findable pillar and covers four things: reachability, a no-JavaScript read window, a fast server response, and being indexable. If any one fails, the best content on the page cannot be cited.
- →Crawler access is the on/off switch for AI citation. Reachability, server rendering, response speed and indexability are all scored under Findable.
- →ChatGPT, Perplexity and Claude do not run JavaScript, so your answer and schema must be in the raw HTML.
- →The Agent-ready tab is separate and advisory. It reads whether an AI agent could act on your page, and it is not part of your score.
What does crawler access mean for AI citation?
Crawler access is the set of conditions that let an AI engine fetch and read your page in the first place. Before any of your content, schema or authorship can help you, the engine has to reach the page, read it, and do so quickly enough to use it at answer time. Rubric scores four crawler-access signals, all under the Findable pillar:
| Signal | The question it answers |
|---|---|
| Reachability | Can the AI crawlers actually fetch the page, past robots.txt and the firewall? |
| No-JavaScript read window | Is the answer, and the schema, in the raw HTML that a non-rendering crawler sees? |
| Fast server response | Does the page respond quickly enough that a crawler does not time out or deprioritise it? |
| Indexable | Is the page free of a noindex rule that excludes it from citation? |
These are gatekeepers. A page can be well written, well sourced and clearly attributed and still score poorly if a crawler cannot reach or read it.
Can the AI crawlers reach your page?
Reachability is whether the citation-serving bots can fetch your page at all. It has two layers: your robots.txt rules, and the firewall or WAF in front of your site. A robots.txt that allows a bot means nothing if the firewall returns a 403 to it, so Rubric pairs your robots rules with a live probe of the citation bots.
The bots that matter are the ones the six engines use: OAI-SearchBot and ChatGPT-User, PerplexityBot, Googlebot, Google-Extended, Bingbot and ClaudeBot. A blocked citation bot means that engine literally cannot quote you. A blocked training-only bot, by contrast, is a legitimate content choice that does not stop live-search citation. See allowing AI crawlers for the exact robots.txt rules, and my crawl was blocked if a firewall is stopping Rubric itself.
Can they read it without JavaScript?
Render parity is whether an AI crawler reads the same page a browser does. It matters because the engines differ on JavaScript: ChatGPT, Perplexity and Claude do not run it, Google renders it, and Copilot renders it unreliably. Anything injected by JavaScript, including JSON-LD schema, is invisible to the engines that do not render.
So your answer and your structured data need to live in the raw HTML the server sends, not be built in the browser afterwards. The test is simple: open view-source, not the inspector, and search for your answer text and your schema. If they are not in view-source, a non-rendering crawler cannot see them. Why AI crawlers do not run JavaScript explains the fix in full.
Is your page fast and indexable?
Two more crawler-access signals sit under Findable. A fast server response matters for retrieval because crawlers work to time and volume budgets; a slow first byte makes a page more likely to be dropped or deprioritised when an engine is assembling an answer. The Pages tab shows each page's load time so you can spot the slow ones.
Being indexable means the page carries no noindex rule. A noindexed page is excluded from AI citation regardless of how well the crawlers can reach it, so a stray noindex on a page you want quoted is a quiet, total block. Rubric flags it per page as the Indexable check.
Where does the report show all this?
Crawler access appears in three places in the report:
What is the Agent-ready tab, and why is it not scored?
The Agent-ready tab is a separate, advisory view that is not part of your Rubric score. Where crawler access asks whether an engine can read and cite you, Agent-ready asks a forward-looking question: could an AI agent act on your page, read a machine-readable price, check availability, book or make contact, not just quote it.
It reads signals such as Offer, price and availability schema, a potentialAction, and a contact or booking point, and it flags them on commercial pages only. Rubric keeps it out of the score on purpose, because agentic search is emerging rather than settled, the same honest treatment it gives Grok and llms.txt. Treat it as guidance on where AI shopping and agents are heading, and see the report, tab by tab for how it fits alongside the scored tabs.