What a technical audit is actually for
A technical SEO audit answers one question: can Google find, crawl and index the pages you want people to find? If the answer is no for an important page, nothing you do to its content or links will matter until that is fixed.
This checklist covers crawlability and indexation only. It does not cover ranking factors, content quality or page speed, and it makes no promise that a page which passes every check will rank. Every statement about how Google behaves below comes from Google's own documentation, which we opened and read on 16 November 2026. The sources are listed at the end, with the date Google shows on each page where it shows one.
One framing point before the steps. Google's own guidance says not every URL on a site should be indexed, only the canonical version of each important page. So the goal of the audit is not "everything indexed". It is "everything important indexed, and everything else excluded for a reason you can name".
Step 1: Start with the Page indexing report
In Search Console, the Page indexing report shows how many URLs Google has crawled and indexed, and, for the URLs it has not indexed, a reason. Start here because it tells you where to look.
Three things from Google's help page for this report change how you read it:
- "Not indexed" is not automatically a fault. Google says it is fine for a URL not to be indexed for the right reasons, for example a robots.txt rule you intended, a noindex tag, a duplicate URL, or a 404 for a page you removed.
- Duplicates should not be indexed. Google says your goal is to get the canonical version of every important page indexed, and that a page marked duplicate or alternate is usually a good thing, because it means Google found the canonical page.
- Do not expect 100% coverage or instant indexing. Google says new content can take a few days to be indexed.
Google also says that if your site has fewer than 500 pages, you probably do not need this report, and can instead check key pages with a site: search. For a single page, use the URL Inspection tool: it shows Google's indexed version of the page, lets you test the live URL, and shows the Google-selected canonical in the indexed data.
What to do: list your important pages (home, service pages, top posts, key landing pages). Inspect each. Mark any that are not indexed, and write down the reason Search Console gives.
Step 2: Check robots.txt, and what it cannot do
A robots.txt file tells crawlers which URLs they may access. Google says it is used mainly to avoid overloading your site with requests, and that it is not a mechanism for keeping a page out of Google.
The mix-up that causes real damage:
- A page blocked by robots.txt can still be indexed if other pages link to it. Google says the URL, and potentially anchor text from links to it, can still appear in results, usually without a description.
- Google also says not to block resource files (scripts, styles, images) if their absence makes the page harder for Google to understand.
What to do: open /robots.txt on your domain. Read every Disallow line. For each one, ask: is this blocking something I want crawled (important pages, or scripts and styles needed to understand them)? Is it being used to try to hide a page from search? If so, the right tool is noindex or a password, not robots.txt.
Step 3: Check noindex, and the trap with robots.txt
A noindex rule, set in a robots meta tag or an X-Robots-Tag HTTP header, tells Google to drop the page from search results. Two details from Google's page matter in an audit:
- 1. For noindex to work, the page must not be blocked by robots.txt. If robots.txt blocks the page, Google never sees the noindex rule, and the page can still appear in results. Blocking a page and noindexing it at the same time cancels the noindex.
- 2. Putting noindex inside robots.txt is not supported by Google.
The reverse risk is just as common on live sites: a noindex tag left on a page after a staging or development phase. Google's page suggests two checks: use the URL Inspection tool to see the HTML Googlebot received, and use the Page indexing report to see pages where Googlebot found a noindex rule.
What to do: for every important page that is not indexed, inspect it and confirm there is no noindex rule you did not intend. For every page you do want hidden, confirm it is not also blocked in robots.txt.
Step 4: Check the status codes
The HTTP status code a URL returns decides what Google does with it. From Google's documentation:
- 2xx: the content may be considered for indexing, but Google says a 2xx status does not guarantee indexing. If a 2xx page shows an empty page or an error message, Search Console reports a soft 404.
- 301 and 308: Google follows the redirect and treats it as a strong signal that the target should be processed. 302 and 307 are treated as a weak signal. Google's crawlers follow up to 10 redirect hops by default.
- 4xx (except 429): Google does not index URLs that return them, and removes already-indexed ones over time.
- 5xx and 429: Google's crawlers temporarily slow down. Already-indexed URLs are preserved for a while, but persistent server errors lead to removal from the index.
What to do: crawl your site (or work from your sitemap) and group URLs by status code. Look for important pages returning 4xx or 5xx, long redirect chains, permanent moves implemented as 302, and "page not found" screens that return a 200 (soft 404s).
Step 5: Check canonicals and duplicates
When the same content is reachable at several URLs (tracking parameters, with and without www, sort and filter pages), Google picks one as canonical. Google lists three ways to signal your preference, in order of strength: redirects (a strong signal), rel="canonical" link annotations (a strong signal) and sitemap inclusion (a weak signal). Using more than one makes your preferred URL more likely to be chosen. None of them is required.
The audit points, all from Google's canonicalization guidance:
- Include a self-referential canonical on the canonical page itself.
- Do not give conflicting signals, for example one URL in the sitemap and a different one in
rel="canonical"for the same page. - Do not use robots.txt for canonicalization.
- Do not use noindex to choose between duplicates within your own site. Google recommends
rel="canonical"instead, because noindex blocks the page from Search entirely. - Link internally to the canonical URL, not to duplicates.
- If JavaScript sets the canonical, it must not change it to something different from the canonical in the original HTML.
In the Page indexing report, "Duplicate without user-selected canonical" means Google chose a canonical for you. "Duplicate, Google chose different canonical than user" means Google disagreed with yours. Both are prompts to inspect the URL and see which page Google picked.
Step 6: Check the sitemap
A sitemap tells search engines which pages you consider important. Google says it helps with discovery but does not guarantee that anything in it is crawled or indexed. Google also says you may not need one if your site is small (about 500 pages or fewer), comprehensively linked internally, and has little video or image or news content you want in results.
What to do: confirm the sitemap is submitted in Search Console, lists only canonical, indexable URLs you want in search (not redirects, 404s or noindexed pages), and does not disagree with your canonical tags.
Step 7: Check links and JavaScript rendering
Google says it can generally only follow a link that is an <a> element with an href attribute. Links that rely only on script events, or <span> elements acting as links, may not be extracted. Links inserted by JavaScript are fine if they use that same markup.
For JavaScript sites, Google describes three phases: crawling, rendering and indexing. Pages with a 200 status go to a rendering queue, and Google says server-side or pre-rendering is still a good idea because it makes the site faster for users and crawlers, and not all bots run JavaScript. For single-page apps, Google warns about soft 404s: use a redirect to a URL that returns a real 404, or add a noindex tag to error views.
What to do: use URL Inspection's rendered view on a key template page and confirm the main content and the navigation links are present. Check that menus and "read more" links are real <a href> links.
The checklist on one page
| # | Check | Where to look | Pass looks like |
|---|---|---|---|
| 1 | Important pages indexed | Page indexing report, URL Inspection | Each important page indexed; every non-indexed URL has a reason you can name |
| 2 | robots.txt | /robots.txt | No important pages or needed resources blocked; not used to hide pages |
| 3 | noindex | URL Inspection, page source, headers | No stray noindex; hidden pages are not also blocked in robots.txt |
| 4 | Status codes | Crawl, sitemap URLs | No important 4xx/5xx; short redirect chains; no soft 404s |
| 5 | Canonicals | Page source, URL Inspection | Self-referential canonicals; no conflicting signals; internal links point to canonicals |
| 6 | Sitemap | Search Console, sitemap file | Only canonical, indexable URLs; consistent with canonicals |
| 7 | Links and rendering | URL Inspection rendered view | Content and nav links present as <a href> in rendered HTML |
If every row passes and a page still is not indexed, check the reason Search Console gives. Google describes "Crawled, currently not indexed" as a common status that does not necessarily indicate an issue, and "Discovered, currently not indexed" as a URL Google found but has not crawled yet, typically because crawling it was expected to overload the site and was rescheduled.
For the wider picture, this checklist sits inside a full digital marketing audit, and fixes here only pay off alongside the long-term work in our SEO strategy guide. If you want a second pair of eyes on your own site, our advanced SEO services cover technical SEO.
Sources (as of 16 November 2026)
All opened and read on 16 November 2026. "Page date" is the date shown on the page itself, where there is one.
- Google Search Central, "Introduction to robots.txt" — https:/
/ (page date: last updated 2025-12-10 UTC)developers.google.com/ search/ docs/ crawling-indexing/ robots/ intro - Google Search Central, "Block Search indexing with noindex" — https:/
/ developers.google.com/ search/ docs/ crawling-indexing/ block-indexing - Google Search Central, "How HTTP status codes affect Google's crawlers" — https:/
/ (reached from developers.google.com/developers.google.com/ crawling/ docs/ troubleshooting/ http-status-codes search/ docs/ crawling-indexing/ http-network-errors) - Google Search Central, "How to specify a canonical URL with rel=canonical and other methods" — https:/
/ developers.google.com/ search/ docs/ crawling-indexing/ consolidate-duplicate-urls - Google Search Central, "Learn about sitemaps" — https:/
/ (page date: last updated 2025-12-10 UTC)developers.google.com/ search/ docs/ crawling-indexing/ sitemaps/ overview - Google Search Central, "Make your links crawlable" — https:/
/ developers.google.com/ search/ docs/ crawling-indexing/ links-crawlable - Google Search Central, "Understand the JavaScript SEO basics" — https:/
/ developers.google.com/ search/ docs/ crawling-indexing/ javascript/ javascript-seo-basics - Search Console Help, "Page indexing report" — https:/
/ support.google.com/ webmasters/ answer/ 7440203 - Search Console Help, "URL Inspection tool" — https:/
/ support.google.com/ webmasters/ answer/ 9012289
Frequently Asked Questions
Does blocking a page in robots.txt keep it out of Google?
Why is my page marked "Crawled, currently not indexed"?
Do I need a sitemap?
How long does it take for a fix to show in Search Console?
Published by
Digitaso Media
Digital Marketing Agency
Digitaso Media is a full-stack digital marketing agency helping businesses generate predictable leads and sales through data-driven SEO, paid advertising, and conversion strategy.
About Digitaso Media →