A crawl of a ten-page brochure site gives you a to-do list. A crawl of a marketplace with two million URLs gives you a spreadsheet nobody will ever finish reading. The findings are not wrong. They are just the wrong unit of work. At scale, you do not fix pages. You fix the things that generate pages.
Think in templates, not URLs
A large site usually has a small number of templates behind an enormous number of URLs: a product page, a category page, a listing with filters, a profile, an article. Every technical decision lives in those templates. If the product template emits a self-referencing canonical, every product does. If the filter template appends parameters without a canonical back to the clean URL, every filtered view becomes a duplicate.
So the first thing I do on a big site is not a full crawl. It is a template inventory: which templates exist, roughly how many URLs each produces, and what each one does about title, canonical, robots directives, internal links, structured data and rendering. One template fixed is tens of thousands of pages fixed. I write about the underlying principles in the technical SEO guide.
Rules instead of decisions
Nobody can hand-decide whether each of three million URLs should be indexed. You need rules that the system applies. Which filter combinations get an indexable page? Perhaps single-facet, in-stock, with more than a threshold number of items. Which get a canonical to their parent? Which get noindex? Which should not be linked at all? The same goes for pagination, sorting, search result pages, and user-generated content.
These rules should be written down, agreed with product and engineering, and encoded in the platform. Where I see large sites go wrong is when the rules exist only in an SEO's head, and a new developer builds a new feature without knowing them.
At scale, the crawl report tells you what happened. The log files tell you what Googlebot actually did about it.
Logs are the ground truth
Crawling a site yourself shows you what is possible to crawl. Server logs show you what search engines actually crawled, how often, and what response they got. On large sites those two pictures regularly disagree. I have seen a site where most of Googlebot's requests were going to a parameter pattern that appeared nowhere in the navigation, because an old script was still generating the links in a footer widget on a single template.
You cannot find that in a crawl. You find it by segmenting logs by template and asking which templates get crawled more than they deserve, and which important ones barely get crawled at all. That tells you where crawl budget is being spent and where crawlability is quietly failing.
The release process is a ranking factor
Most large-site SEO disasters are not caused by algorithms. They are caused by deployments. A staging noindex that ships to production. A new framework version that moves rendering to the client. A redirect map that drops a trailing slash and sends every URL through a chain. These are release problems, and the fix is a release process.
- Automated checks on every deploy: robots.txt, key canonicals, indexability of a sample URL per template, status codes on the top pages.
- A pre-release SEO review for anything that touches a template, routing, or rendering.
- Monitoring that alerts on sudden changes in indexed pages, crawl errors or 5xx rates, so a problem is caught in hours rather than at the monthly report.
This is less glamorous than a clever fix, and it is where the value is. At scale, the site is a system that produces pages, and technical SEO means making that system produce good pages by default. The method I use to get there, template inventory first and then rules, logs and release checks, is laid out in my technical SEO audit framework.