To find where Googlebot may be spending crawls on low-value URLs, compare aggregate crawl activity in Google Search Console’s Crawl Stats report with verified Googlebot requests in your server logs. Group log entries by URL pattern, then look for repeated requests to duplicates, faceted or session URLs, errors, and redirects while important pages remain undiscovered or rarely revisited. Crawl budget is a concern mainly for very large or fast-changing sites; a crawl is not a guarantee of indexing or improved rankings.
What crawl budget means—and who needs to investigate it
Google defines crawl budget as the URLs it can and wants to crawl. Its crawl-budget guide describes two sides of the equation:
As an Amazon Associate I earn from qualifying purchases.
- Crawl capacity: how much crawling a host can serve without being overloaded. Google adjusts crawling in response to factors such as latency, response times, 5xx errors, and 429 rate limiting.
- Crawl demand: which known URLs Google considers worth crawling, influenced by factors including URL inventory, duplication, popularity, staleness, quality, relevance, and update patterns.
A healthy, responsive server can remove a capacity constraint, but it does not make Google want to crawl more URLs if demand is low. In Google’s crawling documentation, a site means a unique hostname: www.example.com and code.example.com have separate crawl budgets.
Google treats crawl-budget optimization as an advanced concern, chiefly for large or rapidly changing sites. Its 2026 guidance gives rough applicability estimates of 1 million or more unique pages with moderate weekly updates, or 10,000 or more unique pages changing daily. These are not hard thresholds. Google also identifies a large share of URLs marked “Discovered – currently not indexed” in Search Console as a possible reason to investigate, but does not set a universal percentage. For smaller sites without many rapidly changing pages, Google says a current sitemap and regular checks of the Page Indexing report are generally adequate.
Use the right evidence for each question
| Evidence source | What it shows | Best use | Limitation |
|---|---|---|---|
| Search Console Crawl Stats | Aggregate Google crawl activity, response groups, and host availability | Spot broad crawl trends and host problems | Does not show crawl history filterable by URL or path |
| Site access logs | Requests and responses for individual URLs and URL patterns | Find which URLs Googlebot actually requested, when, and with what response | Requires log access, parsing, and Googlebot verification |
| A site crawler | URLs discoverable through a site crawl, response codes, redirects, and structural issues | Inventory site structure and identify technical problems | A third-party crawl does not prove Googlebot requested those URLs |
| A log analyser | Bot activity and crawled URLs parsed from supported logs | Help inspect larger or complex log files | Tool support, scale, and cost vary; confirm details with the vendor |
Start with Search Console’s Crawl Stats report for the aggregate picture, then use logs to investigate specific URL patterns. Google says Search Console is available at no cost and reports how much Google has crawled and why. Third-party crawlers and log analysers can help with their respective tasks, but they supplement rather than replace Google’s reports and your server logs. For example, Screaming Frog SEO Spider inventories discoverable URLs, while its Log File Analyser parses supported server logs. Those are vendor-described features, not independent evaluations.
A step-by-step crawl-waste diagnosis
1. Establish whether crawl allocation is a plausible issue
Review Crawl Stats, Page Indexing, and URL Inspection in Search Console. Note host-availability warnings and patterns such as “Discovered – currently not indexed,” but do not treat a single report status as proof that crawl budget is the cause. Google’s crawl troubleshooting guidance lists several possible reasons important URLs may not be crawled or indexed, including poor discovery, blocking, server capacity, crawl prioritization, and quality or demand.
2. Check aggregate crawling and host health
In Crawl Stats, inspect crawl activity, response groups, and host availability. Compare any availability warnings and failing URLs with your own uptime and performance incidents. If Google’s crawling appears constrained while important pages remain underserved, assess whether the host is reaching its serving limit. Additional capacity can allow more requests when capacity is genuinely the constraint; it cannot create crawl demand.
3. Find URL-level history in access logs
Search Console does not expose crawl history filtered by URL or path. Access logs can show when a URL was requested and what response the server returned. Before counting a request as Googlebot, verify it: a crawler can copy Googlebot’s user-agent string. Google recommends reverse DNS verification or checking the published IP ranges in its Googlebot documentation.
Rank #3
4. Group requests and compare them with URLs you want crawled
Group verified requests by path and parameter pattern, then compare their distribution with sitemap URLs and your business-priority pages. Useful groups include canonical landing pages, product or article pages, parameter variations, session IDs, pagination, redirects, error responses, and obsolete URLs. Look for recurring requests to patterns that do not provide unique, useful content while important pages are undiscovered, blocked, slow, or seldom revisited. Google does not prescribe a universal crawl-waste percentage, so this is an operational comparison, not a pass/fail metric.
5. Investigate low-value URL patterns
Google identifies duplicate content, faceted navigation, session identifiers, soft 404s, hacked pages, infinite spaces or proxies, and low-quality or spam content as crawl-efficiency concerns. Faceted navigation can generate huge numbers of URL combinations, and crawlers may request many of them before learning that they are not useful. See Google’s faceted-navigation guidance before choosing how to handle filter URLs.
Rank #4
6. Check technical friction
Review host latency and time to first byte, 5xx and 429 responses, redirect chains, rendering time, and whether Googlebot can access the content and resources it needs. A faster, healthier host can help Google crawl more efficiently, but loading low-value URLs faster does not make their content valuable.
Recommended Free Tools
7. Apply the narrowest appropriate fix and monitor
- Consolidate duplicate URLs where appropriate, and keep the sitemap current with URLs intended for search. Use
lastmodonly when it accurately reflects a meaningful update. - Provide crawlable links to important pages. Return 404 or 410 for content that has been permanently removed, and shorten redirect chains.
- Use robots.txt for URL classes that should not be crawled. Scope and test rules carefully so they do not block valuable pages or resources needed for rendering.
- Recheck logs and Search Console after changes; do not assume a directive will immediately shift requests to another URL group.
Choose directives by the outcome you want
| Mechanism | What it does | Important limitation |
|---|---|---|
robots.txt |
Controls whether crawlers may fetch matching URLs | Blocking is not a removal or deindexing guarantee; a blocked URL may still be known or appear in results |
noindex |
Asks Google not to include a crawled page in the index | Google must fetch the page to read the directive, so it does not save that initial crawl |
| Canonical signals | Communicate a preferred URL among duplicates | They do not prevent crawling; for faceted URLs Google describes canonical and nofollow signals as less effective over the long term than preventing crawl when the combinations should not appear in Search |
| 404 or 410 response | Indicates that a URL does not serve a page | Use for genuinely missing or permanently removed content, not as a substitute for deciding how useful pages should be handled |
For faceted URLs, make the indexation decision first. If filtered combinations should not appear in Search, Google recommends preventing their crawl with robots.txt. If combinations should be crawlable and indexable, normalize parameter order, avoid duplicate filters, use standard separators, and return real 404 responses for empty or nonsensical combinations.
Google does not process the non-standard robots.txt crawl-delay rule, according to its robots.txt specification. Repeatedly toggling robots.txt to free crawl budget for another folder is also not a reliable tactic: Google says newly available capacity will not shift elsewhere unless crawling is already hitting the site’s capacity limit. And do not use prolonged 503 or 429 responses as routine crawl management; Google describes them as temporary emergency responses to an overloaded server and warns extended use can slow crawling or lead to URLs being dropped.
Keep crawl, indexation, and ranking separate
Google Search has distinct crawling, indexing, and serving stages. Googlebot can fetch a page without Google indexing it, and crawling does not guarantee that a page will appear in results. Google’s crawling myths guidance says crawling is needed for search inclusion but is not a ranking signal. As Google’s How Google Search works documentation puts it: “Google doesn’t guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.”
The practical goal is to help Google discover and efficiently revisit valuable URLs, while limiting crawl paths that produce no useful unique content—not to promise a ranking lift from a higher crawl rate.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
