Google indexing: why pages aren't indexed and how to fix it

Google handles every URL in four stages.

1. Discovery. Google finds the URL through a link from a page it already knows, a sitemap or a redirect.
2. Crawling. Googlebot fetches the page, within the limits of robots.txt and how quickly your server responds.
3. Rendering and indexing. Google renders the page (JavaScript included), works out what it is about, picks a canonical URL from any duplicates, and decides whether to store it.
4. Serving. Only indexed pages can appear in results.

Step three is where people get caught out. A crawl is not a promise to index. Google's own documentation says it does not guarantee to crawl, index or serve any page, even one that follows every guideline. A new site, or one that publishes lots of similar pages, will nearly always have some URLs sitting outside the index.

You can see this in the Page indexing report in Google Search Console (Indexing > Pages). It splits your known URLs into indexed and not indexed, with a reason for each group that isn't. Those reasons matter because each one points to a different fix. The two that confuse people most are "Crawled - currently not indexed" and "Discovered - currently not indexed". They sound alike and mean quite different things.

One sense check before you chase anything: a site:yourdomain.com search is a rough sample, not a count. Use Search Console for real numbers.

This status means Googlebot fetched the page, looked at it, and decided not to add it to the index for now. Nothing technical is blocking it. Google has judged that the page isn't worth storing yet, or that another page already covers the same ground.

Common causes

• Thin or unfinished content. Short pages, placeholder copy, tag and archive pages with a few lines of text.
• Near duplicates. Pages that differ only by a town name, a product variant or a URL parameter.
• Weak internal linking. A page linked only from the sitemap looks unimportant.
• Little demand. A page on a topic nobody searches for can sit here for a long time.
• A young site. Google is more selective until a site has built up some trust.

How to fix it

1. Open the URL in URL Inspection and check that the live test renders the real content, not an empty shell waiting on JavaScript.
2. Make the page clearly more useful than what already ranks: answer the question, add original detail, cut boilerplate.
3. Merge or redirect pages that compete with each other, and set a canonical where duplicates have to stay.
4. Link to the page from relevant pages that are already indexed, using descriptive anchor text.
5. Request indexing once, or wait for the next crawl.

If a page stays in this state after real improvements, ask whether it needs to exist at all. Pruning or merging weak pages often helps the rest of a site get indexed.

Here Google knows the URL exists but hasn't crawled it yet. Google's documentation gives the typical reason: it went to crawl, expected the site to be overloaded, and rescheduled. In practice it often means Google doesn't yet see enough reason to spend crawling effort on those URLs.

Common causes

• Server capacity. Slow responses, timeouts or 5xx errors make Googlebot back off. Check Settings > Crawl stats in Search Console for response times and host problems.
• Too many low-value URLs. Faceted navigation, calendar pages, session IDs and endless parameter combinations soak up crawling.
• Orphan pages. URLs that appear only in a sitemap, with no internal links pointing at them.
• A sudden spike. Hundreds of new URLs published at once on a small site.

Crawl budget, briefly

Google says crawl budget mainly matters for very large sites (around a million unique pages or more) or sites with 10,000 or more pages that change daily. If your site has a few hundred pages and still shows many "Discovered" URLs, the problem is usually quality or internal linking, not budget.

How to fix it

1. Fix server errors and slow responses first.
2. Stop generating useless URL patterns, or block them in robots.txt.
3. Link to the pages you care about from pages that are already indexed.
4. Keep your sitemap to canonical, indexable URLs with accurate lastmod dates.

URL Inspection, the search bar at the top of Search Console, is the most direct tool you have. For any URL on a verified property it shows:

• whether the URL is on Google and, if not, why
• the last crawl date and which Googlebot fetched it
• the canonical you declared and the canonical Google selected
• whether crawling and indexing are allowed (robots.txt, noindex)
• structured data and other enhancements Google detected

Click Test live URL to fetch the page as it is right now, then View tested page to see the rendered HTML and a screenshot. This is how you catch a page that looks fine in your browser but renders empty for Googlebot, or a noindex tag a plugin slipped in.

Request indexing puts the URL in a priority crawl queue. It helps for a single important page after a fix. It does not force indexing, it has a daily quota, and pressing it again for the same URL does nothing extra. It is not a way to index a whole site.

There is also a URL Inspection API. It is read-only: it returns the same inspection data so you can monitor pages in bulk, but it cannot request indexing. Tools promising to "instantly index" any page through Google are usually misusing the Indexing API covered in the next section.

Google offers no public API to request indexing of an arbitrary page. The Indexing API exists, but Google limits it to pages with JobPosting structured data or a BroadcastEvent embedded in a VideoObject. Using it for blog posts or product pages is outside its supported use, and Google's documentation warns that abuse can get access revoked.

That leaves the sitemap as the supported route:

• List only canonical, indexable URLs that return a 200 status.
• Keep lastmod accurate. Google uses it when it is consistently correct, and ignores priority and changefreq.
• Submit the sitemap in Search Console (Indexing > Sitemaps) and reference it in robots.txt with a Sitemap: line.
• After a significant change, resubmit it. The Search Console API has a sitemaps.submit method for exactly this.

Google retired its old sitemap "ping" endpoint in 2023, so scripts that still call it do nothing.

A sitemap is a hint, not an instruction. It helps Google find URLs and see what changed; whether a page is indexed still depends on the page.

OpenHelm's SEO agents work this way. A technical SEO job reads index coverage from Search Console, fixes what it can in the product's own repository, and once the fix is live it resubmits the sitemap through Search Console rather than pretending it can force indexing. See SEO with OpenHelm.

IndexNow is an open protocol, launched in 2021 by Microsoft Bing and Yandex, that lets a site tell search engines the moment a URL is added, updated or deleted. You host a key file on your domain, then send the changed URLs to an IndexNow endpoint. One submission is shared with every participating engine, including Bing, Yandex, Naver, Seznam and Yep.

Google does not use IndexNow. It said in 2021 that it would test the protocol and has not adopted it since. Sending IndexNow pings will not speed up Google indexing.

It is still worth setting up, for two reasons:

• Bing matters beyond Bing. Microsoft Copilot runs on Bing's index, and ChatGPT search draws partly on Bing, so fast Bing indexing can help you show up in AI answers sooner.
• It is cheap. Many CMSs, SEO plugins and CDNs support it with a setting, and Cloudflare can send the pings for you.

A sensible order after publishing or updating a page:

1. Make sure the page is in your sitemap with a fresh lastmod.
2. Send the URL through IndexNow for Bing and the others.
3. Let Google pick it up from the sitemap and internal links, and use Request indexing only for the pages that matter most.

For AI search more broadly, see our guides to ranking in ChatGPT and Google AI Overviews.