My crawl was blocked
A blocked crawl means Rubric's crawler was stopped before it could read your pages, almost always by your own firewall, a bot-protection service, or a robots.txt rule. Rubric crawls from GoGoChimp's hardware as a normal web client, so the fix is to let that traffic through, then re-run the crawl.
- →A block comes from your firewall, a bot-protection or WAF service, or a robots.txt disallow, not from Rubric.
- →Rubric crawls as a normal web client from GoGoChimp's hardware, so allow-list that traffic or relax the challenge.
- →Fix the block, clear any cache, then re-run the crawl.
Why do crawls get blocked?
Rubric crawls your site the way a browser or an AI engine would, requesting pages over the public web from GoGoChimp's hardware. Most sites let this through. A crawl gets blocked when something on your side refuses those requests. The usual causes are:
- A firewall or WAF (such as a Cloudflare or similar rule) that challenges or drops automated traffic.
- Bot protection that serves a JavaScript challenge or CAPTCHA instead of the page.
- Rate limiting that rejects requests once they arrive quickly.
- A robots.txt disallow that tells well-behaved crawlers not to fetch some or all paths.
Rubric also refuses to crawl private or internal hosts by design, so a site that only resolves on an internal network cannot be audited. The signal in all of these is the same: an engine that hits the same wall cannot read your pages either, so a block is worth clearing whether or not you are auditing.
How do I let the Rubric crawler through?
Allow-list the crawl traffic, then retry. Because Rubric reads what engines read, the same allowances you would make for AI search bots generally work here too.
What does a robots.txt block look like?
robots.txt is a file at the root of your site that tells crawlers which paths they may fetch. A disallow rule stops well-behaved crawlers, including Rubric, from reading the matched paths. For example:
User-agent: *
Disallow: /That rule blocks every path for every crawler. To allow a crawl while keeping other rules, make sure the paths you want audited are not disallowed. The related article on allowing AI crawlers covers the specific rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot.
I have allowed it and it still fails
If a crawl still will not run after you have allowed the traffic, work through these:
- A cached challenge. Your CDN or bot service may still be serving a challenge from cache. Clear it, then retry.
- An internal or private URL. Rubric will not crawl private or internal hosts. Audit the public URL of the site.
- The whole site is heavily protected. Some sites return only partial pages behind protection. See the related article on failed or partial crawls for what a partial report still tells you.