Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →A small Node.js crawler can turn a bounded set of pages into a useful first-pass audit: collect each page’s response, metadata, headings, directives, images and links, then show which checks failed and why. The 20 checks and scoring weights below are one practical proposal—not an official Google checklist or a prediction of rankings. The approach fetches and parses HTML; it is not a browser renderer or a substitute for Googlebot.
What this crawler can—and cannot—tell you
A compact crawler is useful when you need a repeatable snapshot of page-level issues across a site you control. It can record HTTP status and response duration, titles and descriptions, headings, canonical URLs, robots directives, viewport and language metadata, images without an alt attribute, and links. Comparing those snapshots also reveals patterns such as duplicate titles or descriptions.
As an Amazon Associate I earn from qualifying purchases.
Keep its scope clear. A fetch-and-parse script inspects the HTML it receives; it does not automatically inspect the page after a browser executes JavaScript. Google describes separate crawling, rendering and processing stages for JavaScript pages. Its documentation also notes that JavaScript-injected robots directives can be missed when Google sees a noindex directive and skips rendering. See Google’s JavaScript SEO basics.
Free tools Windows power users keep installed
One-click scans. No signup required.
Likewise, a crawler’s warning is a triage signal, not proof of a ranking penalty. Google recommends descriptive titles and meta descriptions, crawlable links, sitemaps and textually visible content, but that guidance does not validate the cutoffs or score below. See Google’s crawling and indexing documentation.
#1 Best Overall
Separate crawl access from indexability
These controls answer different questions. robots.txt tells crawlers which paths they may request. A noindex directive tells a crawler that can access a page not to include it in search results. A crawler blocked from requesting a page cannot read a robots meta tag on that page or its response headers, so blocking a URL in robots.txt is not a reliable way to communicate that it should be removed from the index.
Google explains that robots meta settings can only be read when the crawler can access the page, and that robots.txt governs crawl access. For keeping an accessible page out of Google, use a noindex directive or authentication rather than treating a robots.txt disallow as an index-removal instruction. See Google’s robots.txt introduction and its robots meta tag guidance.
The 20 checks and their severity
This checklist follows the 20-check proposal by DEV Community author swiftkit_dev. The thresholds are author-chosen heuristics for prioritizing review; they are not universal standards or established Google ranking thresholds. Character counts, word counts, response times and file sizes can help identify pages to inspect, but none alone establishes a ranking outcome.
Errors: 15 points each in the example score
- Missing title: the page has no
<title>element. - Noindex directive: the page has
noindexin a robots meta element or anX-Robots-Tagresponse header. Confirm that the directive is intentional before treating it as a defect. - Broken internal destination: a link points to a page returning a 4xx or 5xx status.
Warnings: 6 points each
- Title length: fewer than 15 or more than 60 characters.
- Missing description: no meta description is present.
- No H1: the page has no level-one heading.
- Duplicate title: another crawled page has the same title. This is a cross-page finding, not something one page can establish alone.
- Image alt missing: an image lacks an
altattribute. Whether it needs descriptive text depends on the image’s purpose; decorative images may appropriately use an empty value. - Short body: fewer than 200 words of extracted text.
- Slow response: the server response takes longer than 1.5 seconds in this crawl.
- Viewport missing: no mobile viewport meta tag is present.
- Plain HTTP: the page uses HTTP rather than HTTPS.
Notices: 2 points each
- Description length: fewer than 50 or more than 160 characters.
- Multiple H1s: more than one level-one heading appears.
- Duplicate description: another crawled page has the same meta description.
- Canonical: no canonical is present, or it points to a different URL. A different target may be intentional; inspect it rather than assuming it is wrong.
- Language missing: the document has no
langattribute. - Open Graph missing: expected Open Graph tags are absent.
- Large HTML: the HTML response exceeds 1 MB.
- Redirect: the requested URL redirects.
Google recommends descriptive metadata and warns that multiple or conflicting canonical tags can produce unexpected results; it does not endorse the specific character cutoffs or severities in this list. See Google’s canonicalization guidance.
Build a bounded crawl in Node.js
For a small audit, start from the home page, fetch it, extract same-site links, and queue eligible URLs until the queue is empty or a page limit is reached. A practical implementation can include subdomains if that matches your definition of “same site,” skip non-HTML files such as PDFs, JPGs and ZIPs, read robots.txt, identify its crawler honestly, and use a small concurrency limit. These are operating choices: set them to fit the site and avoid creating unnecessary load.
- Define the boundary. Choose the starting URL, whether subdomains count, which paths or file types to skip, and a maximum number of pages. Normalize URLs so fragments and equivalent URL forms do not create duplicate visits.
- Read robots.txt and set an identifying user agent. Treat crawl access separately from page-level
noindexchecks; a blocked page cannot be inspected for its meta or header directives. - Fetch with modest concurrency. Record the requested URL, final URL, status, redirect information, elapsed time and response headers. A request timeout and page limit prevent a crawl from running indefinitely.
- Parse HTML and extract fields. The example approach uses Node’s built-in
fetchand thelinkedomHTML parser. Read title, description, H1 elements, canonical, robots metadata, image alt attributes, viewport, document language and anchor links. Combine robots meta directives with theX-Robots-Tagheader; the latter is a response header, not part of the HTML document. - Queue eligible links. Extract destinations from anchor elements with
hrefattributes, resolve relative URLs against the page URL, then apply the crawl boundary and file exclusions before adding them to the queue. - Run cross-page checks after fetching. Group results by title and description to find duplicates; page-level parsing alone cannot reveal those site-wide patterns.
Google generally treats links as crawlable when they use an <a> element with an href, and recommends concise, descriptive, relevant link text. A crawler can flag missing destinations or weak link markup for review, but should not label every finding a ranking penalty. See Google’s guidance on crawlable links.
Check link responses without overwhelming the site
When a link was not fetched during the main crawl—or when you choose to check external destinations—collect candidate URLs and check them once at the end. One practical approach is to try HEAD first, then retry with GET if the server rejects HEAD with 405 or 403. Server behavior varies, so cap the number of requests, use modest concurrency and treat unusual responses as results to verify rather than definitive proof that a destination is broken.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsUse a score readers can recompute
The proposed hygiene score starts at 100 for each page and subtracts points for findings:
Best Value
| Severity | Deduction per finding | Checks |
|---|---|---|
| Error | 15 points | 3 |
| Warning | 6 points | 9 |
| Notice | 2 points | 8 |
For example, one error, two warnings and one notice produce a score of 100 − 15 − 12 − 2 = 71. Show the findings alongside the number so a reader can reproduce it and decide what matters in context. The arithmetic is transparent, not validated science: it does not estimate Google rankings, and the author describes it as a “hygiene score, not a guess at Google’s ranking.”
For site-level reporting, keep page findings and cross-page patterns distinct. A duplicate title is not attributable to a single page in isolation, so report the affected URLs as a group. In the article’s author-reported example, a crawl of 30 pages from a well-maintained open-source documentation site found 27 titles over 60 characters, 24 pages sharing one description, 10 pages sharing a title (described as versioned copies of one documentation page), one page without an H1, two thin pages and no broken links; the reported average score was 90. These are results from that stated example, not independent benchmarks.
When this approach is enough
A small crawler is a good fit when you need a bounded, inspectable audit, want control over request limits, or need to understand exactly how a score was produced. Consider a broader audit tool if you need browser-based JavaScript rendering, larger crawl controls, richer exports, or extensive reporting. Compare options by crawl scope and request controls, rendering, breadth and clarity of checks, duplicate detection, export needs, and whether scoring is visible and configurable. A simple fetch-and-parse script should not be presented as equivalent to a rendered browser audit or Googlebot.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
