Almost nobody reading this has a crawl budget problem, and I would rather say that at the top than sell you five hundred words of suspense.
Google publishes the thresholds. Their large site owner’s guide to managing crawl budget says it is worth thinking about for sites with one million or more unique pages whose content changes about weekly, or ten thousand or more pages changing daily. They describe those as rough classifications rather than hard cutoffs, which is fair, but the order of magnitude is the point.
Most of the sites I am called into have between two hundred and ten thousand pages and update monthly. They are not in either bracket and they are not close.
And yet crawl budget comes up in almost every conversation, usually as the explanation for why pages are not indexed. So it is worth being precise about what it actually is, and about the one version of this that is real.
Two things wearing one name
Google splits it in two, and the distinction is the whole of the practical advice.
Crawl capacity limit is how much Google is willing to ask of your server: how many parallel connections, held how long, before it decides it is being a nuisance. This one responds to how you perform. Slow responses and server errors bring it down. It is a technical ceiling.
Crawl demand is how much Google actually wants your pages. Google names the inputs: perceived inventory, popularity, and staleness. This one responds to whether the pages are worth having.
When somebody says “crawl budget” they almost always mean the first and almost always have a problem with the second. Google is not straining to keep up with your six hundred pages. It has looked at them and is not in a hurry.
Two sites that looked like crawl budget problems and were not
A fitness certification provider: 1,265 indexable pages. Inside that, 1,082 blog posts carrying an unnecessary /blog/ segment, 266 paginated pages with no canonical, 138 pages under 500 words, and a run of author archives several pages deep for authors with a handful of posts each.
A classic car marketplace: 9,220 pages indexed, most of them expired vehicle listings nothing linked to.
Both sites had the symptom people attribute to crawl budget: a large gap between pages that exist and pages doing anything. Neither was anywhere near a million pages. Neither was near ten thousand daily-changing ones. The crawl budget explanation was available, comforting, and wrong in both cases.
What was actually true is duller. The extra URLs were manufactured by the CMS, nobody had decided to publish them, and Google had correctly declined to invest in them. That is not a budget being exhausted. That is Google crawling the pages and choosing not to index them, which is a different status with a different fix.
The version that is real
There is one, and Google names it in the same document: if a significant share of your URLs sit in Discovered, currently not indexed, crawl budget is worth looking at.
Note which status that is. Not crawled and not indexed, but discovered and not indexed. Google knows the URL exists and has not spent the resource to fetch it. That is a queue problem rather than a quality judgement, and on a genuinely large site with a lot of manufactured URLs it is the thing crawl budget describes.
Even then the fix is not “increase the budget”, because you cannot. The fix is to stop generating URLs that consume it. Google’s own list of what wastes it is worth reading in full: faceted navigation, duplicate content, soft 404s, hacked pages, infinite spaces, and low-quality or spam content. Every item is something the site is producing, not something Google is withholding.
What I do instead of raising it
Three things, and none of them is a crawl budget project.
- Stop making the URLs. Turn off the archives nobody browses. Decide, per facet, which filtered views deserve to exist. This is upstream of everything and it is the only fix that stays fixed.
- Use
noindexrather than a robots block on the ones already out there. This is the mechanism people reach for last and should reach for first. HTTP Archive’s 2025 Web Almanac found meta robots tags on 47.9% of mobile pages, but anoindexdirective on only 2.4%. Half the web is emitting the tag and almost nobody is using it to make a decision. Blocking in robots.txt does not remove a page, because Google cannot read a noindex on a URL it is not allowed to fetch. - Look at what Googlebot is actually spending its time on. Search Console, Settings, Crawl Stats, then the breakdown by file type. On more than one site I have found the majority of requests going to scripts and stylesheets rather than pages. That is a real crawl efficiency problem, it has nothing to do with a budget being too small, and it is two clicks from the report everyone is already looking at.
When I do raise it
Honestly? On large ecommerce catalogues with faceted navigation, and on listings platforms with a feed behind them. That is close to the whole list. On those, the URL space is combinatorial, since every filter combination is a page, and it can genuinely outrun what Google will fetch.
If you are a local business, a clinic, a trades company, a professional practice or a store with a few hundred products, and somebody has told you your problem is crawl budget, ask them which of Google’s two thresholds you are near. It is a fair question and it usually ends the conversation.
The short version
- Google’s own thresholds are 1M+ pages weekly or 10,000+ daily. Most sites are not close.
- Capacity limit is about your server. Demand is about whether the pages are worth having. People mean the first and have a problem with the second.
- “Crawled, currently not indexed” is not a budget problem. Google read the page and declined.
- “Discovered, currently not indexed”, at volume, might be. That is the one status worth the conversation.
- You cannot raise it. You can stop spending it.
- Use noindex, not a robots block. Only 2.4% of pages carry one and it is the right tool here.
- Check the crawl stats by file type before concluding anything about volume.
If a site should be ranking and it isn’t, that’s the work I do. Technical SEO covers the crawling and indexing layer. If you’re not sure which of several plausible problems is costing you, that’s what an SEO audit is for.
