Crawl Budget

For Google, crawl budget is the set of URLs on a hostname that its crawling systems can and want to crawl. It reflects both crawl capacity—the connections and time Google can use without overloading the host—and crawl demand, which is influenced by known URL inventory, update needs, perceived value, relevance, and other factors. It is not a fixed page allowance, and a crawled URL is not guaranteed to be indexed.

Why it matters: Most small or slowly changing sites do not need crawl-budget optimisation. Google directs its advanced guidance mainly to very large sites, sites with at least about 10,000 rapidly changing URLs, and sites with many URLs marked “Discovered – currently not indexed”; these are rough indicators, not exact thresholds. If the issue is relevant, use Search Console's Crawl Stats and Page Indexing reports plus server logs to identify the crawlers, URLs, responses, and timing involved. Consolidate unnecessary duplicate or faceted URL inventory, keep sitemaps and meaningful `lastmod` values accurate, return `404` or `410` for permanently removed URLs, remove soft errors and long redirect chains, and keep servers responsive without persistent `5xx` or `429` responses. Diagnose discovery, crawling, canonicalisation, and indexing separately.