Skip to main content
Advanced SEO

Technical SEO Audit Checklist: Can Google Find, Crawl and Index Your Pages?

Most technical SEO problems come down to one question: can Google reach your important pages, and does it keep them? This is a seven-step crawlability and indexation checklist, built only from Google's own Search Central and Search Console documentation, with the common mix-ups (robots.txt versus noindex, 'not indexed' versus 'broken') spelled out.

DM
Digitaso Media·Digital Marketing Agency·October 31, 2026·8 min read
Technical SEO Audit Checklist: Can Google Find, Crawl and Index Your Pages?

What a technical audit is actually for

A technical SEO audit answers one question: can Google find, crawl and index the pages you want people to find? If the answer is no for an important page, nothing you do to its content or links will matter until that is fixed.

This checklist covers crawlability and indexation only. It does not cover ranking factors, content quality or page speed, and it makes no promise that a page which passes every check will rank. Every statement about how Google behaves below comes from Google's own documentation, which we opened and read on 16 November 2026. The sources are listed at the end, with the date Google shows on each page where it shows one.

One framing point before the steps. Google's own guidance says not every URL on a site should be indexed, only the canonical version of each important page. So the goal of the audit is not "everything indexed". It is "everything important indexed, and everything else excluded for a reason you can name".

Step 1: Start with the Page indexing report

In Search Console, the Page indexing report shows how many URLs Google has crawled and indexed, and, for the URLs it has not indexed, a reason. Start here because it tells you where to look.

Three things from Google's help page for this report change how you read it:

  • "Not indexed" is not automatically a fault. Google says it is fine for a URL not to be indexed for the right reasons, for example a robots.txt rule you intended, a noindex tag, a duplicate URL, or a 404 for a page you removed.
  • Duplicates should not be indexed. Google says your goal is to get the canonical version of every important page indexed, and that a page marked duplicate or alternate is usually a good thing, because it means Google found the canonical page.
  • Do not expect 100% coverage or instant indexing. Google says new content can take a few days to be indexed.

Google also says that if your site has fewer than 500 pages, you probably do not need this report, and can instead check key pages with a site: search. For a single page, use the URL Inspection tool: it shows Google's indexed version of the page, lets you test the live URL, and shows the Google-selected canonical in the indexed data.

What to do: list your important pages (home, service pages, top posts, key landing pages). Inspect each. Mark any that are not indexed, and write down the reason Search Console gives.

Step 2: Check robots.txt, and what it cannot do

A robots.txt file tells crawlers which URLs they may access. Google says it is used mainly to avoid overloading your site with requests, and that it is not a mechanism for keeping a page out of Google.

The mix-up that causes real damage:

  • A page blocked by robots.txt can still be indexed if other pages link to it. Google says the URL, and potentially anchor text from links to it, can still appear in results, usually without a description.
  • Google also says not to block resource files (scripts, styles, images) if their absence makes the page harder for Google to understand.

What to do: open /robots.txt on your domain. Read every Disallow line. For each one, ask: is this blocking something I want crawled (important pages, or scripts and styles needed to understand them)? Is it being used to try to hide a page from search? If so, the right tool is noindex or a password, not robots.txt.

Step 3: Check noindex, and the trap with robots.txt

A noindex rule, set in a robots meta tag or an X-Robots-Tag HTTP header, tells Google to drop the page from search results. Two details from Google's page matter in an audit:

  • 1. For noindex to work, the page must not be blocked by robots.txt. If robots.txt blocks the page, Google never sees the noindex rule, and the page can still appear in results. Blocking a page and noindexing it at the same time cancels the noindex.
  • 2. Putting noindex inside robots.txt is not supported by Google.

The reverse risk is just as common on live sites: a noindex tag left on a page after a staging or development phase. Google's page suggests two checks: use the URL Inspection tool to see the HTML Googlebot received, and use the Page indexing report to see pages where Googlebot found a noindex rule.

What to do: for every important page that is not indexed, inspect it and confirm there is no noindex rule you did not intend. For every page you do want hidden, confirm it is not also blocked in robots.txt.

Step 4: Check the status codes

The HTTP status code a URL returns decides what Google does with it. From Google's documentation:

  • 2xx: the content may be considered for indexing, but Google says a 2xx status does not guarantee indexing. If a 2xx page shows an empty page or an error message, Search Console reports a soft 404.
  • 301 and 308: Google follows the redirect and treats it as a strong signal that the target should be processed. 302 and 307 are treated as a weak signal. Google's crawlers follow up to 10 redirect hops by default.
  • 4xx (except 429): Google does not index URLs that return them, and removes already-indexed ones over time.
  • 5xx and 429: Google's crawlers temporarily slow down. Already-indexed URLs are preserved for a while, but persistent server errors lead to removal from the index.

What to do: crawl your site (or work from your sitemap) and group URLs by status code. Look for important pages returning 4xx or 5xx, long redirect chains, permanent moves implemented as 302, and "page not found" screens that return a 200 (soft 404s).

Step 5: Check canonicals and duplicates

When the same content is reachable at several URLs (tracking parameters, with and without www, sort and filter pages), Google picks one as canonical. Google lists three ways to signal your preference, in order of strength: redirects (a strong signal), rel="canonical" link annotations (a strong signal) and sitemap inclusion (a weak signal). Using more than one makes your preferred URL more likely to be chosen. None of them is required.

The audit points, all from Google's canonicalization guidance:

  • Include a self-referential canonical on the canonical page itself.
  • Do not give conflicting signals, for example one URL in the sitemap and a different one in rel="canonical" for the same page.
  • Do not use robots.txt for canonicalization.
  • Do not use noindex to choose between duplicates within your own site. Google recommends rel="canonical" instead, because noindex blocks the page from Search entirely.
  • Link internally to the canonical URL, not to duplicates.
  • If JavaScript sets the canonical, it must not change it to something different from the canonical in the original HTML.

In the Page indexing report, "Duplicate without user-selected canonical" means Google chose a canonical for you. "Duplicate, Google chose different canonical than user" means Google disagreed with yours. Both are prompts to inspect the URL and see which page Google picked.

Step 6: Check the sitemap

A sitemap tells search engines which pages you consider important. Google says it helps with discovery but does not guarantee that anything in it is crawled or indexed. Google also says you may not need one if your site is small (about 500 pages or fewer), comprehensively linked internally, and has little video or image or news content you want in results.

What to do: confirm the sitemap is submitted in Search Console, lists only canonical, indexable URLs you want in search (not redirects, 404s or noindexed pages), and does not disagree with your canonical tags.

The checklist on one page

#CheckWhere to lookPass looks like
1Important pages indexedPage indexing report, URL InspectionEach important page indexed; every non-indexed URL has a reason you can name
2robots.txt/robots.txtNo important pages or needed resources blocked; not used to hide pages
3noindexURL Inspection, page source, headersNo stray noindex; hidden pages are not also blocked in robots.txt
4Status codesCrawl, sitemap URLsNo important 4xx/5xx; short redirect chains; no soft 404s
5CanonicalsPage source, URL InspectionSelf-referential canonicals; no conflicting signals; internal links point to canonicals
6SitemapSearch Console, sitemap fileOnly canonical, indexable URLs; consistent with canonicals
7Links and renderingURL Inspection rendered viewContent and nav links present as <a href> in rendered HTML

If every row passes and a page still is not indexed, check the reason Search Console gives. Google describes "Crawled, currently not indexed" as a common status that does not necessarily indicate an issue, and "Discovered, currently not indexed" as a URL Google found but has not crawled yet, typically because crawling it was expected to overload the site and was rescheduled.

For the wider picture, this checklist sits inside a full digital marketing audit, and fixes here only pay off alongside the long-term work in our SEO strategy guide. If you want a second pair of eyes on your own site, our advanced SEO services cover technical SEO.

Sources (as of 16 November 2026)

All opened and read on 16 November 2026. "Page date" is the date shown on the page itself, where there is one.

Frequently Asked Questions

Does blocking a page in robots.txt keep it out of Google?
No. Google says a page blocked by robots.txt can still be indexed if other pages link to it, though it usually appears without a description. To keep a page out of results, use noindex or password protection.
Why is my page marked "Crawled, currently not indexed"?
Google says this means the page was crawled but not indexed, and that it is a common message that does not necessarily point to a problem. Google does not index every page it crawls. Inspect the URL, check the checks above, and look at whether the page is a duplicate or adds little beyond other pages on your site.
Do I need a sitemap?
Google says you might not if your site is small (about 500 pages or fewer), well linked internally, and has few media or news pages. For larger or newer sites, it can help Google discover pages, though it does not guarantee crawling or indexing.
How long does it take for a fix to show in Search Console?
For a "Validate fix" request, Google says validation typically takes up to about two weeks, and in some cases much longer. For a new page, Google says indexing can take a few days.
DM

Published by

Digitaso Media

Digital Marketing Agency

Digitaso Media is a full-stack digital marketing agency helping businesses generate predictable leads and sales through data-driven SEO, paid advertising, and conversion strategy.

About Digitaso Media →

Ready to Put This into Practice?

Get a free growth audit from Digitaso Media. We will identify exactly where your biggest opportunities are — delivered within 48 hours, no obligation.

Get Your Free Audit