Crawl budget and faceted navigation

Crawl budget is the amount of crawling Google is willing to do on your site in a given period. For most sites it is not a limit you will ever reach. For sites with filters, sorting, pagination and search parameters, it is the reason two-thirds of your pages are never indexed — because Googlebot spent its visit on color-and-size combinations instead.

What crawl budget is made of

Google describes it as the product of two things. Crawl capacity is how much your server can take: Googlebot watches response times and errors, and backs off when it sees the site struggling. A slow server directly reduces how much gets crawled, which is one of the practical reasons server response time matters beyond user experience.

Crawl demand is how much Google wants to crawl: driven by how popular your URLs are, how often they genuinely change, and how much value the site has demonstrated. A site whose pages never change gets crawled less, correctly.

You can raise capacity with a faster server. You raise demand by being worth crawling. Neither is a setting.

How faceted navigation explodes

A category page with five filters — size, color, brand, price band, in stock — where each can be applied independently and in any order, does not produce five extra URLs. It produces every combination, in every ordering, multiplied by every sort option, multiplied by every page of pagination.

Three hundred genuine product pages become several hundred thousand crawlable URLs, nearly all of them showing subsets of the same items. Every one is a link Googlebot can follow, and it will.

The symptom in Search Console is distinctive: a very large “Discovered — currently not indexed” count, a crawl stats report dominated by URLs with query strings, and important pages that take weeks to be picked up. The pages you care about are not blocked — they are queued behind hundreds of thousands of URLs that should never have existed.

The two tools, and why using both breaks everything

This is the distinction that decides whether your fix works, and it is the one most commonly got wrong.

robots.txt prevents crawling. Googlebot never fetches the URL, so it costs nothing. But a URL it cannot fetch is a URL whose contents it cannot read — including any noindex tag on it. Blocked URLs with inbound links can still appear in results, without a description.

noindex prevents indexing. It reliably keeps a page out of the index, and it consolidates nothing — but Googlebot has to crawl the page to see the tag. So noindex does not save crawl budget at all.

Which means: never apply both to the same URL. If robots.txt blocks it, the noindex is invisible and the page can stay indexed indefinitely. This combination is deployed constantly by people trying to be thorough, and it produces the exact outcome they were trying to prevent.

The rule of thumb: use robots.txt for URLs that were never supposed to be indexed and are not, and noindex for URLs that are indexed and should not be — then, once they have dropped out, block them if crawl volume is still a problem.

Fixing facets, in order of preference

Do not generate the URLs. The strongest fix by a wide margin. Apply filters client-side without changing the URL, or handle them via POST, or write them into a fragment. No URL means no link, no crawl and no duplicate. This is a development decision rather than an SEO configuration, which is why it gets skipped in favor of weaker options.

Block the parameter patterns in robots.txt. Practical when the URLs already exist and were never indexed. Block the parameters, not the paths:

Disallow: /*?*sort= and Disallow: /*?*filter= rather than blocking /category/, which would take your real pages with it. Test every rule in Search Console’s robots.txt tester before deploying — an over-broad Disallow is the single fastest way to de-index a site.

Canonicalise filtered views to the unfiltered page. Works for indexing, does nothing for crawling — Google still fetches every URL to read the tag. Use it alongside blocking, not instead of it.

Keep a small number of valuable facets indexable. Some filtered views have genuine search demand: “waterproof hiking boots” is a query, “waterproof hiking boots size 9 sorted by price ascending” is not. Pick the handful with demand, give them clean static URLs and real intro copy, and treat them as landing pages. Block the combinatorial remainder.

Mark facet links rel="nofollow". The weakest option. It discourages discovery through those links but Google may still find the URLs elsewhere. Do not rely on it alone.

The other budget drains

Facets are the biggest, not the only one.

Redirect chains. Every hop is a separate fetch. A three-hop chain across ten thousand URLs is thirty thousand requests to deliver ten thousand pages.

Soft 404s and thin pages. Empty search results, out-of- stock products, tag archives with one item. Crawled repeatedly, worth nothing.

Session IDs and tracking parameters in internal links. Every unique parameter value is a unique URL. If your own site appends campaign parameters to internal links, you are generating duplicates of your own pages.

Infinite spaces. Calendars with next-month links forever, “load more” endpoints that never end. Googlebot will keep going.

Stale sitemaps. Listing URLs that redirect or 404 spends crawl on nothing. And if your lastmod dates are all today’s date because they are generated at build time, they carry no information and Google will stop trusting them.

Measuring it

Search Console → Settings → Crawl stats is the only first-party source. Look at total requests per day, average response time, and the breakdowns by response code, file type and purpose. The “By purpose” split between discovery and refresh is the informative one: heavy discovery crawling with little refresh means Googlebot is spending its time finding new URLs rather than revisiting your real pages.

Server logs are better if you can get them, because they show every Googlebot request rather than an aggregate. Sort by URL pattern and the drain is usually obvious within minutes.

Start by checking what you are allowing

Our robots.txt checker shows what your current rules actually permit, which is worth confirming before adding more of them — over-broad Disallow lines are common and invisible. The sitemap checker catches stale entries wasting crawl on redirects and 404s. For the indexing side of this, Search Console page indexing errors explains the statuses these changes will move, and auditing a large website covers doing this work at a scale where you cannot inspect pages one at a time.