Most indexing problems I get called in for are not mysterious. They are the predictable result of a site that makes Google work too hard to find the pages that matter, and too easy to find the pages that don't. This sub-guide of my technical SEO guide is about fixing that balance.
Crawl budget, honestly
Crawl budget is the number of URLs Googlebot is willing and able to fetch from your site in a given period. It is shaped by how fast your server responds without errors (crawl capacity) and how much Google wants your content based on popularity and freshness (crawl demand).
For a 300-page business site, crawl budget is almost never the constraint. For a marketplace with millions of URLs, a publisher with decades of archives, or a programmatic SEO build, it is often the single biggest technical lever.
The practical test: compare the URLs Google crawls per day (Search Console crawl stats) against the URLs you want indexed. If the first is far smaller, or most of the crawl goes to URLs you don't care about, you have a crawl budget problem. If neither is true, fix something else.
Three controls, three different jobs
Robots.txt, noindex and canonical tags get mixed up constantly. Each answers a different question.
| Control | What it tells Google | Stops crawling? | Stops indexing? | Passes link signals? | Use it when |
|---|---|---|---|---|---|
| robots.txt Disallow | "Do not fetch this URL." | Yes | No. A blocked URL can still be indexed from links, usually without a snippet. | No (Google can't see the page's links) | Saving crawl on URLs that should never be fetched: internal search, infinite filters, admin paths. |
| Meta robots noindex | "Fetch this, but keep it out of the index." | No. Google must crawl it to see the tag. | Yes | Initially; long-term noindex pages are eventually treated as nofollow | Pages users need but that shouldn't rank: thank-you pages, thin tag archives, filtered views with no demand. |
| Canonical tag | "This URL duplicates that one; consolidate signals there." | No | Usually, if Google agrees | Yes, consolidated to the canonical | Genuine duplicates: parameter variants, tracking URLs, printer versions, syndicated copies. |
The classic mistake is blocking a URL in robots.txt and adding noindex to it. Google can't crawl the page, so it never sees the noindex, so the URL sits in the index indefinitely. If you want something out of the index, let Google crawl it. If you want to save crawl, block it and accept it may linger as a bare URL. I go deeper on the third control in the canonical tags guide.
Crawl traps: where budget goes to die
A crawl trap is any URL pattern that generates a practically infinite number of crawlable URLs with little unique content. I find at least one on most mid-sized sites, and several on ecommerce sites.
Faceted navigation
Filters for colour, size, price, brand and sort order multiply against each other. A category with ten filters of five values each can produce millions of URL combinations, each a page that looks almost identical to its neighbours. The fix is a deliberate policy: decide which facet combinations have real search demand ("cotton kurtas under 1000" might; "cotton kurtas sorted by price, page 3" does not), make those crawlable and self-canonical, and keep the rest out via robots.txt rules on parameters or by not rendering them as crawlable links. The site architecture guide covers the design side.
Calendars and date archives
Event and booking sites often expose a "next month" link forever, and Googlebot will follow it into the year 2090. Cap the range and stop linking beyond a sensible horizon.
Session IDs and tracking parameters
Any parameter that changes the URL without changing the content (session tokens, UTM tags on internal links, affiliate IDs) creates duplicates. Strip them from internal links, canonicalise the clean version, and add robots.txt rules for the worst offenders. A related trap is the broken relative link that produces recursive paths like /blog/page/page/page/.
Orphan pages: the opposite problem
An orphan page has no internal links pointing to it. Google might know about it from a sitemap or an old external link, but with no internal path it gets crawled rarely, carries almost no signals, and tends to fall into "Discovered – currently not indexed". On large sites, orphans often include valuable pages: products whose category was removed, guides never added to a hub, location pages generated by a template but never linked from navigation.
Finding them means comparing two lists: every URL from a crawl (what's linked) against every URL from sitemaps, logs, analytics and the CMS (what exists). Anything in the second list but not the first is an orphan. I ran a controlled test on this in the indexing orphan pages experiment, and the internal linking guide covers reconnecting them systematically.
Index bloat
Index bloat is thousands of low-value URLs that Google has indexed: tag pages with one post, paginated comment pages, empty categories, staging subdomains, HTTP and HTTPS both live, parameter variants. It matters because Google assesses sites partly on aggregate quality. A site where most indexed pages are thin looks like a thin site, and the good pages suffer by association.
Compare "indexed" in the Search Console page indexing report with the count of pages you actually want indexed. A large gap means bloat. Fixing it means deciding, pattern by pattern, whether each group should be improved, consolidated with canonicals, noindexed, or removed with a 410.
Diagnosing the two "not indexed" statuses
These two Search Console statuses sound the same and mean different things.
Discovered – currently not indexed
Google knows the URL exists but has chosen not to crawl it yet. This is a crawl problem, not (directly) a quality problem. Common causes: crawl capacity is being spent elsewhere; the URL is only reachable from a sitemap with no internal links; the server was slow or erroring when Google tried; or the site's overall quality is low enough that Google is in no hurry. Fixes are about prioritisation: internal links from strong pages, removing crawl traps, faster server responses, and trimming the sitemap to URLs you actually want.
Crawled – currently not indexed
Google fetched the page and decided not to index it (for now). This is a quality or duplication judgement. Common causes: thin content, a near-duplicate of another indexed URL, a template page with nothing unique, rendered content Googlebot couldn't see, or a fine page sitting in a cluster of weak ones. Fixes are about the page and its neighbours: make it meaningfully different, consolidate duplicates, strengthen internal links, and check the rendered HTML in URL Inspection to confirm Google sees the content.
What log files tell you that nothing else can
Search Console shows a sample. Crawlers show what could be crawled. Server logs show what Googlebot actually fetched, how often, and with what status code. Above a few thousand URLs, logs are the only way to answer: which sections does Google crawl daily and which monthly? What share of crawl goes to parameter URLs or 404s? Are new pages fetched within days or weeks?
You need raw access logs (or a CDN export), filtered to verified Googlebot IPs, grouped by URL pattern and status code. The analysis is not complicated; getting the logs usually is, so I put it in the audit scope early. The process is in my technical SEO audit framework.
How I prioritise
Confirm the money pages are indexed
Are the pages that should earn traffic actually in the index? Check the template groups that matter (categories, products, service pages, core guides) with URL Inspection and a site search. If they are missing, nothing else matters yet.
Cut the biggest waste
Find the single largest crawl trap or bloat pattern from logs and the indexing report and fix that one pattern. On most sites this frees more crawl capacity than everything else combined.
Reconnect orphans and deepen links to weak sections
Use the crawl-versus-sitemap comparison. Add hub and contextual links so valuable pages sit within a few clicks of the homepage.
Resolve conflicting signals
Canonicals to noindexed URLs, blocked or redirecting URLs in the sitemap, canonicals pointing at redirects. Each conflict makes Google guess.
Then improve speed and server health
Faster, error-free responses raise crawl capacity. It's a multiplier on everything above, but rarely fixes an indexing problem on its own.
All of this is implementation work, not recommendations. Crawlability improves when the robots.txt actually changes, the links actually get added and the templates actually get fixed. That is what I mean by building SEO systems rather than delivering reports.
Frequently asked questions
Does a small website need to worry about crawl budget?
Almost never. Crawl budget becomes a real constraint at tens of thousands of URLs or more, or when crawl traps generate huge numbers of junk URLs. For a small site, focus on indexability, internal linking and content quality instead.
Why is a page I blocked in robots.txt still showing in Google?
Robots.txt stops crawling, not indexing. If other pages link to the blocked URL, Google can index the URL without ever fetching it. To remove a page from the index, unblock it, add a noindex tag, and let Google recrawl it.
How long until "Discovered – currently not indexed" pages get indexed after fixes?
It depends on how often Google crawls the site. After adding internal links and removing crawl waste, I usually see movement within a few weeks on actively crawled sites, and longer on sites Google visits rarely. Resubmitting the sitemap and requesting indexing for a few representative URLs can help.
Should I noindex thin pages or delete them?
If the page has no value to users either, delete it and return a 410 (or 404) so Google drops it and stops crawling it. If users need the page but it should not rank, noindex it. Noindexing hundreds of pages nobody visits just preserves crawl waste.
Is submitting a sitemap enough to get pages indexed?
No. A sitemap helps Google discover URLs, but discovery is not indexing. Pages with no internal links and nothing distinctive tend to stay in "Discovered" or "Crawled – currently not indexed" regardless. Internal links and genuine content do the heavy lifting.
Where this connects
This guide is one part of the technical SEO hub. The systematic checks are laid out in my technical SEO audit framework; the orphan-page mechanics were tested in the indexing orphan pages experiment; the technical SEO and content architecture case study shows what it looks like on a real site; and if you want this done rather than described, see my technical SEO services.