A first-person report from Toolore describes inheriting a domain with hundreds of thousands of old URLs. Its author added a robots.txt rule to block query-string URLs, but the early Search Console numbers did not show that the change had fixed indexing. The case is useful as a diagnosis—not as proof that the same rule is safe or effective on every site.
How a 66-page site ended up with 957,000 known URLs
In a September 24, 2026 post on DEV Community, Toolore’s author said that six weeks after launching a 66-page static Next.js site, Google Search Console showed 957,000 known URLs. The sitemap, by contrast, listed 66 pages. These are figures from the author’s Search Console view, not an independently audited count or a general benchmark.
As an Amazon Associate I earn from qualifying purchases.
The domain had previously been an e-commerce store. The author found old query-string URLs that now returned the new site’s homepage with HTTP 200, even though those URLs did not identify real pages on the new site. The post reported 621,728 URLs as “Crawled – currently not indexed,” 335,238 as “Soft 404,” 358 as “Not found (404),” 65 as “Discovered – currently not indexed,” and one indexed page.
Those categories describe different situations. The author’s diagnosis was that many obsolete query-string URLs produced homepage responses that looked like duplicates or missing pages. Some old product paths, however, already returned genuine 404 responses; the author left those alone.
#1 Best Overall
Why a homepage response can become a soft 404
Google defines a soft 404 as “a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code.” In other words, the HTTP response says success while the page itself signals that the requested resource is absent or irrelevant. Google may classify that mismatch as a soft 404. Google’s crawling-error guidance explains the distinction and how to handle such URLs.
Returning the homepage for an unrelated, obsolete product or search URL does not make that URL a meaningful replacement. It can create confusing responses for visitors and crawlers. A useful first step is to inspect representative examples in Search Console and group them by URL pattern; a sitemap marked “Success” does not explain why URLs in the Pages report are excluded.
What the robots.txt rule did in this case
The post’s example file included this rule:
User-Agent: *
Allow: /
Allow: /icon.svg
Disallow: /*?
Sitemap: https://toolore.com/sitemap.xmlThe disallow pattern targets URLs containing a question mark, so it blocks crawling of query-string URLs. The author said the site had no pages that depended on query parameters. The explicit allow for /icon.svg was included because the favicon URL used a cache-busting query string.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →This is a site-specific configuration, not a safe default to copy. Before blocking a pattern, check whether real pages, filters, sorting options, or assets use it. Google’s URL structure guidance discusses dynamic URLs, including generated search results and sorting or filtering spaces, as cases where crawl restrictions may be worth considering. A broad rule can also prevent Google from crawling useful URLs if a site relies on query parameters.
Choose the fix for the outcome you need
Robots.txt controls crawling; it is not a reliable way to remove a URL already known to Google from search results. Google says a blocked URL can still be indexed if other pages link to it. If a page should remain accessible but not appear in Google Search, Google must be able to crawl it to read its noindex directive. See Google’s robots.txt guide and noindex guidance.
| Situation | Appropriate response | Important distinction |
|---|---|---|
| Stop crawling a problematic URL pattern | Consider a robots.txt rule after checking the site’s real pages and assets. | Blocking crawling does not guarantee removal from search results. |
| Keep a page accessible, but exclude it from Google Search | Use a crawlable noindex directive. |
If robots.txt blocks the URL, Google may not be able to see the directive. |
| Content is permanently gone and has no comparable replacement | Return an HTTP 404 or 410. | Do not redirect an irrelevant URL to the homepage. |
| A removed page has a clear, relevant replacement | Use a permanent redirect to that replacement. | Redirect only when the destination genuinely serves the old page’s purpose. |
Google’s crawl-budget guidance also cautions against expecting an automatic benefit from blocking old URLs: “Blocking or hiding already crawled pages from recrawls won’t shift your crawl budget to another part of your site unless Google is already hitting your site’s serving limits.” Google further notes that blocked URLs can remain in its crawl queue longer and may be recrawled if the block is removed.
Check the site before changing the rule
- Inspect examples. In Search Console’s Pages report, examine sample URLs from each major status category. Group them by path and query-string pattern rather than assuming all reported URLs have the same cause.
- Test what each URL serves. Determine whether it is a real page, a missing resource, or an irrelevant URL returning the homepage with HTTP 200. Keep genuine 404s distinct from soft 404s.
- Inventory legitimate query-string use. Check pages, filters, sorting, and assets such as cache-busted files. Do not block a pattern until you know what on the current site depends on it.
- Apply the matching fix. Use robots.txt to restrict crawling, crawlable
noindexto exclude accessible content, 404 or 410 for content with no replacement, and a redirect only for a clear substitute. - Review the results over time. Search Console counts can lag. Compare later observations, but do not treat a change in one category as proof of a ranking recovery or of the rule’s causal effect.
What the early follow-up did—and did not—show
Toolore’s author said it was too early to claim success because Search Console data lagged by several days. At the time of the post, “Discovered – currently not indexed” had moved from 65 to 55, while the indexed count remained at one. That is a preliminary, self-reported snapshot; it does not establish that the robots.txt change caused the movement, that rankings recovered, or that other sites should expect the same timeline.
The author also reported changing the sitemap’s lastmod values from a shared build time to each page’s content update date. Google recommends using lastmod to reflect when an indexed URL changed, while noting that a sitemap is a hint, not a guarantee of immediate crawling or indexing. Google’s crawling-error and sitemap guidance describes those limits.
Best Value
The post mentions submitting URLs through IndexNow, but the available account does not establish that this produced faster discovery. That claim should not be treated as a demonstrated outcome of the fix.
Using a temporary removal request
Google’s Removals tool can temporarily hide information from Search, but its effect lasts about six months. For lasting removal, the underlying URL needs an enduring solution, such as a suitable status code or a crawlable noindex, depending on the situation. Google explains the distinction in Remove Your Site Info from Google.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
