Googlebot Crawl Budget Allocation for Large Sites

Crawl budget for large sites comes down to two separate levers: how much crawling your server can handle without falling over, and how much crawling Google actually wants to do. Get either one wrong and Googlebot burns its attention on pages nobody searches for while your product pages sit undiscovered. This piece walks through what changed in Google's guidance over the past year, who this problem actually applies to, and what to do about it if it applies to you.

Google's own definition treats separate subdomains as separate budgets: www.example.com and shop.example.com don't share a pool. The formula underneath is simple to state and harder to manage: crawl budget equals the lower of crawl rate limit and crawl demand. Whichever one is more restrictive sets the ceiling. Crawl capacity is the supply side, essentially how much connection time Google's crawlers can hold against your server without causing problems. Google calls this hostload internally. Every site starts at the same cautious default, and Google raises that ceiling automatically once the site proves it can handle more, watching mainly response time, error rates, and the appearance of 429s from the server. Crawl demand is the other side of the equation: Googlebot's own judgment about how much revisiting a site's URLs is worth its time, shaped by perceived URL inventory, popularity (tied loosely to PageRank), how often content changes, and how unique that content is. Of those four signals, perceived inventory is the one site owners can actually move. If Google thinks a site has more URLs than it really does, crawl attention gets spread thin across pages that don't deserve it.

What changed in Google's official guidance in the July 2026 documentation rewrite

On July 22, 2026, Google rewrote its large-site crawl budget documentation from the ground up. The old URL now forwards, via a 301 redirect, to /crawling/docs/crawl-budget. Anyone with that older page bookmarked will land somewhere new.

The terminology shifted too. What the original 2017 explainer called "crawl rate limit" is now "crawl capacity limit," and the Search Console crawl rate setting that older guide referenced doesn't exist anymore. Small change on paper, but it matters if you're auditing a site against documentation someone printed out three years ago.

The bigger addition is the idea of a shared capacity pool. Google's new language states: "While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others." That single sentence changes how site owners should think about crawl planning. AdsBot and the Google Shopping crawler draw against the same pool as Googlebot. For a large e-commerce account with an active Merchant Center feed, that means feed crawling is competing directly with the crawl requests that would otherwise go toward discovering new or updated product pages.

Which sites need to manage crawl budget (and which problems look like crawl budget but aren't)

John Mueller has said that "crawl-budget is over-rated. Most sites never need to worry about this." That's the right filter to apply before doing anything else in this piece. Crawl budget management is not a general best practice that every site should adopt; it's a fix for a specific, fairly narrow set of circumstances.

Google's own thresholds define that set: sites with more than a million unique pages that get updated at least weekly, sites with more than 10,000 unique pages where content changes daily, and sites where a large share of URLs sit in Search Console's "Discovered, currently not indexed" bucket. The million-page threshold hasn't moved since 2020, and Gary Illyes confirmed as much again in 2025. It's a stable number, not a moving target.

What that means in practice: for something like 99% of business websites, the thing that looks like a crawl budget problem is actually a content quality problem, a weak internal linking structure, or a slow server. Those three things produce symptoms that resemble crawl budget trouble (pages not getting indexed, stale cache dates, thin organic visibility), but the fix is entirely different. Throttling crawl rate or blocking sections in robots.txt won't help a site whose real issue is thin, duplicate, or poorly linked content. Diagnosing the wrong problem here wastes real engineering time.

Gary Illyes's 2025 database-speed insight and how it reshapes fix prioritization

Crawl budget conversation has, for years, centered almost entirely on page count: more pages, more crawl demand, more to manage. In May 2025, on the Search Off the Record podcast, Gary Illyes pushed back on that framing directly.

His point was about cost, not volume: "If you are making expensive database calls, that's going to cost the server a lot." A site running 500,000 pages against slow, expensive SQL queries can create more crawling trouble than a site serving 2 million pages of static, cached HTML. Page count was never the real variable. Server cost per request was.

That tracks with how Googlebot actually behaves on a first pass. It acts as a plain HTTP client, measuring server response time, something close to time-to-first-byte, rather than the full rendered load time a browser experiences. JavaScript execution happens later, in a separate rendering queue that runs on its own delayed schedule. Google's documentation points to a target: TTFB under 300 to 400 milliseconds on average. Sites that clear that bar comfortably tend to get crawled more freely; sites that don't, tend to hit a ceiling regardless of how much content they publish.

Documented cases exist of crawl rate roughly quadrupling, from around 150,000 URLs to 600,000, purely by cutting TTFB from 800 milliseconds down to 180. Documented cases exist of crawl rate roughly quadrupling, from around 150,000 URLs to 600,000, purely by cutting TTFB from 800 milliseconds down to 180. No new content, no restructured navigation, no change to the sitemap. Just a faster server response, and Google rewarded it with more crawl attention.

What the February 2026 2 MB HTML crawl limit means for page architecture

On February 3, 2026, Google reorganized its crawler documentation and, in the process, spelled out file size limits that had previously lived mostly in developer folklore. Googlebot processes the first 2 MB of HTML for Search. PDFs get 64 MB. Other Google crawlers default to a 15 MB ceiling.

The mechanism matters here: this limit applies to the raw source HTML, before any JavaScript runs through the Web Rendering Service. Googlebot fetches partially, stops right at the 2 MB mark, and hands whatever it downloaded off to indexing as though that were the whole page. No error gets thrown. No warning appears in Search Console. The truncation is silent.

John Mueller was quick to frame this as documentation catching up to reality: "None of these recently changed, we just wanted to document them in more detail." The 15 MB number that floated around in older guides referred to fetching, he clarified, not to how much of that fetched content indexing actually processes.

Practically, this rarely bites. The median HTML page runs around 33 KB on mobile, something like 60 times under the 2 MB ceiling. Even at the 90th percentile, pages are around 151 KB. For nearly all pages, the 2 MB limit is a non-issue. It does start to matter on pages that inline enormous JSON payloads, sprawling inline scripts, or bloated structured data blocks directly into the HTML source, a pattern that appears on large product listing pages or pages built with heavy server-side rendering that dumps state into the markup.

The URL inventory problem: how large sites drain crawl budget before Googlebot reaches what matters

Across a set of more than 35 crawl budget audits, 38% of the URLs Googlebot was actively crawling on e-commerce sites generated zero organic traffic and carried no real search intent behind them. That's more than a third of crawl activity spent on pages that were never going to earn a click, resources that should have gone toward product and category pages instead.

Google states the consequence directly: "If Google spends too much time crawling URLs that it shouldn't, Google's crawlers might not explore the rest of your site." Perceived inventory, again, is the lever site owners actually control.

Three patterns account for most of this waste. Faceted navigation and URL parameters are the single largest drain on e-commerce sites: a catalog offering 8 colors, 6 sizes, and 5 sort orders produces 240 possible combinations per product line, and a mid-size catalog of 10,000 products can quietly generate 500,000 or more crawlable URL variants, almost none of which need to exist as separate indexable pages.

Soft 404s are the second pattern: pages that return a 200 OK status but display "out of stock" or "page not found" messaging still get crawled as if they were live, valid content. Gary Illyes confirmed this exact failure mode at Search Central Live Asia Pacific in 2025. Technical duplicate content is the third, and it's larger than most site owners assume: roughly 60% of the web consists of duplicate content, and most of it is technical rather than editorial in origin, HTTP versus HTTPS, www versus non-www, trailing slash inconsistencies, and tracking parameters like?utm_source= or?fbclid= all producing separate crawlable URLs for what is functionally one page.

On status codes, one more distinction applies. A 410 signals permanent removal and gets treated as a harder stop than a 404, cutting off recrawl attempts sooner. For anything deleted for good, 410 is the faster route to getting Googlebot to stop wasting requests on it.

Practical controls for directing Googlebot toward high-value URLs

robots.txt is the blunt instrument, and it works well for blocking whole classes of low-value URLs: filter and facet combinations, internal search result pages, pagination variants, tracking-parameter URLs, and admin, account, cart, and checkout paths, along with API endpoints that don't need to be crawled. One caveat carries real weight here: blocking CSS or JavaScript in robots.txt breaks rendering, so theme and framework assets need to stay reachable no matter what else gets blocked. Robots.txt blocks crawling, not indexing. It blocks crawling, not indexing. Robots.txt blocks crawling, not indexing, so noindex is the right tool for anything that needs to stay crawlable but shouldn't appear in results.

Canonical tags work on the demand side rather than the capacity side. Every page should carry a self-referencing canonical tag, and every parameter variant should point back to the clean version of the URL. Google will still crawl the variant occasionally, but a solid canonical signal tells it, over time, to stop treating that variant as a separate inventory item worth tracking.

Sitemap hygiene sends its own signal. A sitemap should list only indexable URLs, with lastmod dates that are actually accurate, not just refreshed on a schedule regardless of whether the page changed. A sitemap padded with noindexed, redirected, or low-value URLs sends Google a mixed message about what's worth exploring. Sitemap presence tells Google a URL is worth checking out; its absence doesn't block crawling outright, but it does reduce the odds that new URLs get found without a nudge.

304 Not Modified support rounds this out. When a server correctly returns a 304 for a page that hasn't changed, Google reuses its cached copy instead of re-downloading the whole thing. That saves crawl capacity without telling Google to stop caring about the page, which is a distinction robots.txt and noindex can't offer.

The shared crawl capacity pool and the rising AI crawler problem

The shared-pool structure documented in July 2026 has a direct consequence for e-commerce sites specifically. AdsBot and the Google Shopping crawler pull from the same capacity limit as Googlebot itself, and image crawlers likely share that pool too, even though Google's own documentation names only AdsBot and Google Shopping as examples. A Merchant Center feed running heavy, frequent updates is, in effect, competing with Googlebot for the same finite slice of server attention that would otherwise go toward crawling and discovering product pages.

That framing matters more now than it would have a few years back, because the crawler landscape has gotten more crowded. Large sites aren't just managing Googlebot's appetite anymore; they're managing a growing set of automated clients hitting the same infrastructure, each with its own crawl demand and none of them coordinating with each other. Server capacity was always the fixed constraint that produced crawl budget: as more automated clients hit the same infrastructure, each with its own crawl demand and none coordinating with each other, capacity determines how much of that demand can be met. The number of things drawing on that capacity has only gone up.