Four stages, not one
"We're not ranking" is almost never a ranking problem on a small site. It's usually a page that was never fetched, or was fetched and not stored. Those are different failures with different fixes, and treating them as one is why people spend months writing content that no engine has read.
| Stage | What happens | Fails when |
|---|---|---|
| Discovery | The engine learns the URL exists | Nothing links to it and it's in no sitemap |
| Crawling | The URL is fetched | robots.txt blocks it, or the server errors |
| Rendering | JavaScript is executed to see the final page | Content needs a script that failed or timed out |
| Indexing | The page is stored as a candidate result | It's noindex, a duplicate, or judged too thin to keep |
| Ranking | It's ordered against other candidates for a query | This is the only stage that's about quality |
Reading index states
A search console reports why each URL is or isn't indexed. The wording varies by engine, but the categories are consistent, and each points at a different fix.
| State | Means | Fix |
|---|---|---|
| Discovered, not crawled | Known but never fetched | Usually low priority — improve internal links to it |
| Crawled, not indexed | Fetched and judged not worth storing | Content problem: thin, duplicated, or no clear purpose |
| Duplicate, engine chose a different canonical | It found a near-identical page it prefers | Consolidate, or set the canonical you actually want |
| Excluded by noindex | You told it not to index | Intentional? If not, find the stray tag or header |
| Blocked by robots.txt | Not allowed to fetch | Unblock — and remember a block prevents reading a noindex |
| Soft 404 | Returns 200 but looks like an error or empty page | Return a real 404, or put content on it |
The one worth dwelling on is "crawled, not indexed." It's the engine saying it read the page and decided storing it wasn't worth the disk. On a new site a handful of these is normal. Across most of your pages, it's a verdict on the content, and no technical fix will change it.
Crawl budget, and who it applies to
Crawl budget is the rough number of pages an engine will fetch from your site in a period. It's a function of how important your site seems and how fast your server responds.
When it does apply, the cause is almost always URL multiplication: filters and sort parameters producing thousands of near-identical pages. `?colour=blue&sort=price` and `?sort=price&colour=blue` are two URLs with one page behind them.
- Disallow filter and sort parameters in robots.txt, so they're never fetched
- Set a canonical on every filtered view pointing at the unfiltered page
- Return real 404s for combinations that have no results, rather than an empty 200
- Keep the server fast — a slow site is crawled less, which is a compounding penalty
Sitemaps in practice
A sitemap is a list of URLs you want indexed. It's a hint about discovery, not an instruction to index, and it doesn't override anything — a page in your sitemap that's blocked in robots.txt stays blocked.
- Include only canonical, indexable URLs — a sitemap full of redirects and noindex pages is a contradiction the engine has to resolve, usually against you
- Generate it from the same data the pages render from, so it can't go stale
- Set `lastmod` honestly. Stamping today's date on every URL every day makes the field worthless and is treated as such
- Split above 50,000 URLs, and use a sitemap index
- Reference it from robots.txt and submit it in a search console
// Generated from the content collections, so a new lesson appears in// the sitemap without anyone remembering to add it.const learningRoutes = [ "/learning", ...lessons.map((l) => `/learning/lessons/${l.slug}`), ...dictionaryTerms.map((t) => `/learning/dictionary/${t.slug}`),];Diagnosing a missing page, in order
- Search for the exact URL in the engineIf it appears, it's indexed and you have a ranking question, not an indexing one. Stop here.
- Fetch the URL yourself with JavaScript disabledIf the content is missing, the engine's first pass sees what you just saw.
- Check robots.txt and the page's meta robots tagTwo lines, ninety seconds, and the cause of a surprising share of these.
- Check the URL is in your sitemap and linked from somewhereAn orphan page with no internal links is a page the engine has little reason to reach or to value.
- Read the index state in a search consoleBy this point you've ruled out the mechanical causes, and the state tells you whether it's a duplicate or a content judgement.
Common mistakes
- Treating "not ranking" and "not indexed" as the same problem
- Optimising crawl budget on a two-hundred-page site
- Putting noindex on a page that's also blocked in robots.txt, so the noindex is never read
- Sitemaps containing redirects, 404s, and noindex URLs
- Stamping every sitemap entry with today's date, which makes lastmod meaningless
Key takeaways
- Discovery, crawling, rendering, indexing, and ranking are five stages that fail differently
- "Crawled, not indexed" is a content verdict, not a technical bug
- Crawl budget matters above a few thousand URLs, and almost never below
- A sitemap is a discovery hint and a coverage diagnostic, not an indexing instruction
Try it yourself
Open your site's coverage report in a search console and compare submitted URLs against indexed ones. Read the reason attached to the largest excluded group — that one reason usually explains most of the gap.