How to audit a large website

Auditing a 300-page site and auditing a 300,000-page site are not the same activity at different scales. Past roughly ten thousand URLs, you can no longer look at the pages, your crawler will not finish, and the reports return numbers too large to act on. The method has to change from inspection to sampling.

Start by counting URLs you did not know you had

The first genuinely useful number on a large site is the gap between the URLs you intended to publish and the URLs that exist. Compare three counts:

What your CMS says — the number of published items. What your sitemap says — the number of URLs you are asking Google to index. What Search Console knows — the total in the Page indexing report, indexed and not indexed together.

On a healthy large site the first two match closely. When the third is an order of magnitude larger, you have URL generation you did not intend — filters, parameters, session identifiers, pagination without limit. That finding alone frequently explains most of the site’s indexing problems, and it takes ten minutes to establish. See crawl budget and faceted navigation for what to do about it.

Inventory the templates

Group every URL by the template that renders it. On an e-commerce site that is usually: home, category, subcategory, filtered category, product, brand, search results, blog index, blog post, and a handful of utility pages.

For each template, record how many URLs use it, how many are indexed, and which of those URLs get organic traffic. That table is the audit’s spine, and it makes the prioritization obvious in a way a URL list never does: a template with 80,000 URLs and 200 indexed is a very different problem from one with 400 URLs and 380 indexed, even if both look like “indexing issues” in a report.

Then audit three or four URLs per template, thoroughly, in dependency order — the technical audit method applies unchanged, just to a sample. What is true of one product page is almost always true of all of them.

Crawl with limits, deliberately set

An unconstrained crawl of a large site with faceted navigation does not finish. It also produces a report you cannot read. Configure it instead:

Cap the crawl. Set a URL limit and a maximum depth. Depth 4 or 5 covers what matters, because anything deeper is barely crawled by Google either.

Exclude the parameter space — but note what you excluded and how large it was, because that number is itself a finding.

Sample within templates rather than crawling every instance. Most crawlers support this; a few thousand product URLs tell you what 200,000 would.

Throttle it. A fast crawler against a production site is a load test you did not schedule. Rate-limit and, if you can, run it out of hours.

Crawl as a rendered browser only on a sample. Rendering is an order of magnitude slower. Render a sample per template to check for the gap between HTML and DOM, then crawl the rest without it.

Use log files — this is the part that only works at scale

Server logs are the single most valuable and most neglected source for a large site, because they are the only record of what Googlebot actually did rather than what a crawler could do. On a small site there is not enough data to be interesting. On a large one there is more than enough.

Take a month of access logs, filter to verified Googlebot, and answer four questions:

Which URL patterns absorb the crawling? Group requests by template. If 60% of Googlebot’s requests go to filtered category URLs that you do not want indexed, that is your crawl budget problem quantified rather than assumed.

Which important pages are rarely crawled? Cross-reference your revenue or traffic pages against crawl frequency. Pages crawled once a month update in search once a month.

What status codes is Googlebot receiving? Logs show the real distribution, including intermittent 5xx errors that a crawl at 2pm on a Tuesday will never reproduce.

Which crawled URLs are not in your sitemap? These are the URLs you did not know you had, listed by name.

Search Console’s Crawl stats report is a weaker version of the same thing — aggregated, sampled, no per-URL detail — but it needs no access to infrastructure, which sometimes decides the matter.

Prioritize by exposure, not by severity

Severity labels do not survive scale. “Critical: missing canonical” on one obsolete page matters less than “Warning: duplicate title” across every product page. Rank findings by template reach multiplied by the value of the pages affected.

A practical ordering: things suppressing indexing on high-value templates first; things wasting crawl at volume second; things affecting how existing indexed pages appear third; everything else after.

And be explicit about what you sampled and what you did not. An audit that silently covers 2% of the site while reading as though it covers all of it is worse than one that states its sampling, because the reader cannot calibrate the findings.

Watch for the failures specific to scale

Sitemap index files. Above 50,000 URLs or 50MB uncompressed you need multiple sitemaps behind an index. Split them by template rather than arbitrarily — then the indexed-versus-submitted ratio per sitemap becomes a per-template diagnostic for free.

lastmod dates that lie. If every entry shows today because the sitemap is generated at build time, the field carries no information and Google stops trusting it. Either use real modification dates or omit the field.

Internal link distribution. On a large site, link equity concentrates through navigation and pagination in ways nobody designed. Check how many internal links point at your most valuable templates — often strikingly few, while thousands point at pagination.

Programmatic thin content. Templates generating pages from sparse data produce large numbers of near-identical pages. Google samples them rather than indexing them all, which is the usual explanation for a large crawled — currently not indexed count.

Rendering cost at volume. A JavaScript-dependent large site compounds the rendering queue problem, because every level of link depth needs another crawl-render cycle before the next level is even discovered — JavaScript SEO and rendering covers the mechanism.

Where to begin

Get the three URL counts, build the template table, then sample. Our free site check is a useful first pass on one representative URL per template — it covers the tier 1 to 3 mechanics quickly, so you can establish whether a template has a systemic fault before committing to a full crawl. The sitemap checker is worth running against each sitemap in an index separately, since a single bad segment is invisible in the aggregate.