In ten focused minutes, you can determine whether representative pages respond, whether crawl and index controls are blocking them, whether important URLs are discoverable, and whether JavaScript changes what users (or a crawler) can see. Use a small Python check for repeatable HTTP evidence, then Chrome DevTools for network and rendering clues. Finish with Search Console or server logs when you need Google-specific confirmation. This is a triage workflow, not a ranking or indexing guarantee.
What this audit can—and cannot—prove
Google describes three minimum technical conditions for a page to be eligible for indexing: Googlebot must not be blocked, the server must return HTTP 200, and the page must contain indexable content. Passing those checks does not mean Google will index or serve the URL. Google Search Central explicitly notes that “just because a page meets these requirements doesn’t mean that a page will be indexed; indexing isn’t guaranteed.”
As an Amazon Associate I earn from qualifying purchases.
The ten-minute limit is a workflow constraint, not a tested promise that every site can be audited in exactly ten minutes. It works best as a first pass over a few representative URLs. A large site, an outage, or a complicated JavaScript application needs a deeper crawl and log analysis.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Choose representative URLs before starting the timer
Prepare three to five absolute URLs:
- The homepage or another stable entry point.
- An important landing page that should attract search traffic.
- A page type that depends on JavaScript, such as a product, application, or filtered listing.
- A recently changed URL, if one exists.
- Optionally, a URL that has historically produced errors.
Record each URL exactly as published, including HTTPS, host, path, and meaningful parameters. The goal is to compare different templates and failure modes, not to sample dozens of nearly identical pages.
#1 Best Overall
Minutes 0–2: establish reachability with Python
Run a standard-library request against each selected URL. The script below follows normal redirects, reports the final URL, status, response headers, and a small body sample, and distinguishes HTTP errors from connection failures. It does not emulate Googlebot or determine whether a URL is indexed.
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
URLS = [
"https://example.com/",
"https://example.com/important-page/",
"https://example.com/app-page/",
]
for url in URLS:
request = Request(
url,
headers={
"User-Agent": "technical-seo-audit/1.0",
"Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
},
)
try:
with urlopen(request, timeout=10) as response:
sample = response.read(512)
print(f"URL: {url}")
print(f" status: {response.status}")
print(f" final_url: {response.geturl()}")
print(f" content_type: {response.headers.get('Content-Type', '')}")
print(f" x_robots_tag: {response.headers.get('X-Robots-Tag', '')}")
print(f" sample_bytes: {len(sample)}")
except HTTPError as error:
print(f"URL: {url}")
print(f" http_error: {error.code} {error.reason}")
print(f" final_url: {error.geturl()}")
print(f" x_robots_tag: {error.headers.get('X-Robots-Tag', '')}")
except URLError as error:
print(f"URL: {url}")
print(f" connection_error: {error.reason}")
print()urllib.request.urlopen follows redirects by default, so response.geturl() tells you where the request ended. If the final URL is different, note every redirect you can observe in DevTools or your server logs; a redirect is not the same result as a direct 200 response from the requested URL.
Interpret the first response
| Observation | What it tells you | Immediate follow-up |
|---|---|---|
| HTTP 200 and the expected final URL | One minimum technical condition is met. | Inspect directives and content; 200 alone does not establish indexability. |
| 3xx response ending at another URL | The requested address is being redirected. | Check that the destination is the intended canonical URL and that there are no loops or long chains. |
| 4xx response | The server is refusing or cannot find the requested resource. | Verify the path, access rules, authentication, and internal links. |
| 5xx response, timeout, or DNS/TLS failure | A server or network problem may prevent reliable crawling. | Compare with a browser and inspect uptime, hosting, and server logs. |
Run the same URL in a browser if Python fails. A discrepancy can indicate user-agent handling, an intermittent outage, a certificate issue, or a request that requires browser execution. Do not label a page “not indexed” from a failed local request; this step only measures your request.
Recommended Free Tools
Minutes 2–4: inspect crawl and index controls separately
Check robots.txt for crawl rules
Open https://your-domain.example/robots.txt in a browser or request it with Python. Look for a Disallow rule under the relevant user-agent that covers the target path. Also check whether required JavaScript, CSS, image, or API paths are unintentionally disallowed; blocking a resource can change what a crawler can render.
robots.txt controls crawling, not guaranteed removal from search. Google may know a disallowed URL through links and show a result without fetching the page. Therefore, do not use a robots rule as a substitute for noindex or authentication. A crawler must be able to fetch a page to see a noindex directive.
Inspect page-level directives
In the initial HTML, search for a robots meta element such as <meta name="robots" content="noindex">. Then inspect the response headers for X-Robots-Tag, which is especially important for PDFs, images, feeds, and other non-HTML responses. Record the directive, its scope, and the exact URL where it appears.
A page can return 200 while carrying noindex; that combination is reachable but intentionally excluded from indexing. Conversely, a page that is disallowed in robots.txt may prevent Google from reading a newly added noindex. Treat these controls as separate checks and resolve conflicts according to the intended outcome.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Minutes 4–6: verify discovery and sitemap alignment
Trace ordinary crawlable links
From the homepage or another known page, find a normal HTML link to each important URL. Links should use an anchor element with an href containing the destination; links that appear only after an interaction or are assembled in a script deserve additional testing. Follow the path manually and note whether a user can reach the page without relying on an obscure form submission.
Important pages need discoverable, crawlable links. A sitemap can supplement discovery, but it cannot compensate for an otherwise isolated architecture or guarantee a crawl.
Check the sitemap
Open the sitemap location declared in robots.txt or the site’s normal sitemap index. Confirm that it lists the intended absolute canonical URLs, uses the correct protocol and host, and does not fill the inventory with tracking-parameter variants, redirects, blocked URLs, or pages marked noindex. Submission is a hint to Google, not a command to crawl or index every entry.
| Sitemap limit | Required handling |
|---|---|
| 50,000 URLs per sitemap file | Split larger inventories and reference the files from a sitemap index. |
| 50 MB uncompressed per sitemap file | Split by URL count, file size, or logical section before the limit is exceeded. |
For each sampled URL, mark three independent facts: linked from a findable page, present in the sitemap, and intended as canonical. “Present in the sitemap” does not mean “indexed.”
Free tools Windows power users keep installed
One-click scans. No signup required.
Minutes 6–8: compare the response with browser rendering
Use Chrome DevTools Network
- Open the URL in Chrome and press F12 (or choose More tools → Developer tools).
- Select Network, enable Preserve log, and reload the page. Disable cache only while DevTools is open if you need a fresh load.
- Click the document request. Check status, final response headers, redirects, timing, and the Response tab.
- Filter by JS, CSS, Fetch/XHR, and Img. Investigate red requests, 4xx/5xx responses, blocked requests, and requests that never finish.
- Open Initiator and Headers for a failed resource to identify the script, policy, or server response responsible.
Compare the initial response (right-click the document request and open its response, or use View page source) with the live DOM in the Elements panel. Look specifically for main copy, headings, internal links, structured navigation, and canonical or robots elements. If critical text or links exist only after JavaScript executes, the page needs a Google-specific rendering check rather than a conclusion based solely on the browser view.
Understand what rendering means for Google
Google renders JavaScript with a recent version of Chrome during processing, but rendering is a separate stage from crawling and indexing. A browser can display content successfully while Google has not processed it, or while a failed API call leaves the crawler with less content than a user sees. Use the URL Inspection live test or rendered view in Search Console to examine Google’s current fetch and rendered result.
Minutes 8–10: classify findings and choose evidence
Create a short record for every observation. Include the URL, the exact symptom, the evidence (status, header, rule, request, or screenshot), and the next diagnostic action. Use distinct labels:
- Not reachable: the request fails, times out, or returns an error.
- Blocked: a robots rule, authentication layer, firewall, or resource policy prevents crawling.
- Not discoverable: no ordinary crawlable path leads to the URL and it is absent or misrepresented in the sitemap.
- Not indexable by directive: a fetched response contains
noindexor an equivalent header. - Google status unknown: local checks pass, but no Search Console or log evidence confirms what Google fetched or indexed.
For a Google-specific answer, run Search Console’s URL Inspection on the exact URL and use its live test when appropriate. Review the Page Indexing report for patterns across URL groups. Server logs can show requests to particular URLs and user agents; Search Console does not provide a URL-filtered crawl history, so logs are the better source for that question.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesWhich tool answers which question?
| Tool | Best evidence | What it cannot establish alone |
|---|---|---|
Python with urllib.request |
Repeatable status, redirect destination, response headers, and connection errors. | Google’s crawl behavior, rendered state, or index status. |
| Chrome DevTools | Browser network requests, failed resources, redirects, and rendered DOM. | Whether Google indexed or will serve the page. |
| Search Console URL Inspection | Google’s reported URL status and a live fetch/render test. | A complete historical request log for one URL. |
| Server logs | Requests received for specific URLs, times, status codes, and user agents. | Why Google chose to index, rank, or omit a page without additional evidence. |
Common patterns and the safest next move
“It returns 200, so it must be indexed”
HTTP success satisfies only one minimum condition. Check the page’s indexable content and directives, then verify the URL in Search Console.
“robots.txt blocked it, so it is removed”
A blocked URL can remain known to Google through links. Decide whether the goal is crawl control, indexing exclusion, or access protection, and implement the control that matches that goal.
“The sitemap contains it, so Google will crawl it”
Sitemap submission is a discovery hint. Repair internal linking and remove inaccurate entries while keeping only the canonical URLs you intend to expose.
“The page works in Chrome, so Google sees the same thing”
Check the initial HTML, failed network requests, and JavaScript-generated content. Then use URL Inspection for a Google-specific rendering result.
“The audit found a problem, so a fix guarantees rankings”
Technical repairs remove obstacles; they do not guarantee indexing or ranking. Re-test the URL, request validation where available, and monitor logs and Search Console for evidence of change.
A compact worksheet for future audits
Save a row for each URL with these columns:
- Requested URL and final URL
- HTTP status and response time
- Robots rule affecting the path
- Meta robots and
X-Robots-Tag - Canonical URL
- Internal-link path found
- Sitemap presence and accuracy
- Key JavaScript or API requests
- Search Console status or live-test result
- Owner, priority, and next action
Repeat the same worksheet after a deployment or template change. Consistent evidence makes it easier to distinguish a one-off outage from a structural problem and prevents a passing browser test from being mistaken for proof of Google’s indexing state.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
