Understanding a failed or partial crawl
A partial crawl means Rubric scored some of your pages but not all of them, usually because the site hit the crawl's page and time ceiling or because heavy bot protection blocked part of it. A failed crawl means Rubric could not score enough to report. A partial report is still useful: it scores the pages it reached.
- →A partial crawl scored some pages, not all, usually due to the page/time ceiling or heavy bot protection.
- →A failed crawl could not score enough to report at all, usually a full block or an unreachable host.
- →A partial report still tells you plenty: pillar scores and template fixes hold across the whole site.
What causes a partial or failed crawl?
Rubric puts a hard page and time ceiling on every crawl as a guard, so no single job runs forever. A crawl comes back partial or failed when it runs into one of these:
- The page ceiling. On the Free plan a crawl audits up to 500 pages, so a larger site is capped at the first 500. Pro lifts that cap, so a Pro crawl is bounded only by the time ceiling, not a page count.
- The time ceiling. A very slow site can hit the time limit before every page is read.
- Heavy bot protection. A firewall, WAF or CAPTCHA challenge can block some or all pages. If it blocks part of the site, you get a partial report; if it blocks everything, the crawl fails.
- An unreachable or private host. Rubric refuses private or internal hosts and cannot read a site that is down, so the crawl fails.
- A stale job. If a crawl stalls, Rubric reclaims the stale job rather than leaving it hanging, which can surface as a failed run to retry.
Partial versus failed: what is the difference?
| Partial crawl | Failed crawl | |
|---|---|---|
| What happened | Some pages were scored | Too few pages were scored to report |
| Usual cause | Page or time ceiling; part of the site protected | Full block, unreachable host, or private host |
| What you get | A report on the pages reached | No usable report; a prompt to fix and retry |
| What to do | Read it, then widen coverage if needed | Fix the block, then re-run |
What does a partial report still tell me?
A partial report is not a wasted crawl. Rubric scores per page and the site score is the median of the pages it reached, so even a partial set gives you a reliable read on your typical page. More importantly, template-level findings hold across the whole site: if a shared header is missing Organization schema on the pages that were crawled, it is missing everywhere that template renders. So the highest-value fixes, the ones in Phase 01 of the Action Plan, are usually valid site-wide even from a partial crawl.
How do I turn a partial crawl into a full one?
Work through the cause, then re-run: