Crawlability

Crawlability is a crawler's ability to discover a URL, request it, receive a useful response, and access the page resources and rendered content it needs to process. Internal links, sitemaps, robots rules, authentication, DNS and network availability, HTTP status codes, redirects, resource blocking, and client-side rendering can all affect that path. Crawlability is crawler-specific because different services use different user agents, rendering systems, and access rules.

Why it matters: Separate discovery, crawling, rendering, canonicalisation, indexing, and ranking when diagnosing Search visibility. A `robots.txt` disallow can prevent a crawler from fetching content, while `noindex` is an indexing rule that the crawler generally must fetch to see; blocking the same URL in robots.txt can stop Google from reading that rule. Login requirements protect private content rather than merely creating an SEO error. For intended public pages, check internal links, sitemap inclusion, robots behaviour for the relevant verified crawler, redirect destinations, final status, rendered HTML, required resources, canonicals, and accidental robots directives. Use server logs, Search Console's URL Inspection, Page Indexing, and Crawl Stats data. A successful crawl makes indexing possible but does not guarantee it.