Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteA Googlebot user-agent is only a claim; anyone can send it. To verify a request, use the source IP in your server log and either (1) perform Google’s documented reverse-DNS then forward-DNS confirmation, or (2) match the address against the appropriate, current Google crawler CIDR list. An ASN lookup can add ownership context, but an ASN match alone does not prove that the caller is Googlebot.
What you are actually proving
Verification answers a narrow question: does this network request originate from infrastructure Google identifies for the crawler or fetcher category you are checking? It does not prove that every request from a Google-owned network is Googlebot, and it does not authenticate the user-agent header.
- User-agent: Google Search Central warns that the HTTP
user-agentheader used by Googlebot is often spoofed. - Source IP: use the address that actually connected to your server, not an address copied from an untrusted header such as
X-Forwarded-Forunless your trusted proxy has already validated that header. - Category: common crawlers (including Googlebot), special-case crawlers such as AdsBot, and user-triggered fetchers have different purposes and may use different hostnames or range files.
Google’s official overview is What Is Googlebot. Its verification guidance says the best methods are reverse DNS with confirmation or matching the source IP against Google’s published Googlebot ranges.
Get the correct source IP from your logs
Start with the access-log field representing the TCP connection peer. In a typical Nginx or Apache combined log this is the first field, but confirm your format. If a CDN, load balancer or reverse proxy sits in front of the origin, identify which component wrote the log and whether it records the original client address in a trusted, authenticated field.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →#1 Best Overall
- Copy the complete IPv4 or IPv6 address, without brackets, a port, or punctuation.
- Record the timestamp, requested path, response status and user-agent for later correlation.
- Check whether the request passed through a proxy. Never let an arbitrary client-supplied forwarding header determine your allowlist decision.
Do not block or allow a crawler solely from the text Googlebot in the user-agent. Treat it as a lead for investigation.
Manual verification: reverse DNS followed by forward DNS
This is Google’s documented one-off workflow for a few log entries. The two DNS directions are important: reverse DNS establishes a hostname associated with the IP, while forward DNS confirms that the hostname really points back to the same address.
1. Reverse-resolve the IP
On Linux or macOS, run:
host 66.249.66.1Google’s documentation gives 66.249.66.1 as an example that resolves to crawl-66-249-66-1.googlebot.com. Your result should be treated as an example, not as a permanent allowlist entry.
For IPv6, pass the address to the same utility (quote it if your shell requires it). You can also use dig -x ADDRESS +short or the equivalent resolver available on your operating system.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →2. Check the hostname’s domain and pattern
The returned name should belong to the Google domain appropriate to the caller. Google’s verification material describes names under googlebot.com, google.com and googleusercontent.com. A common Googlebot hostname often contains googlebot.com; other crawler and fetcher categories can use different documented patterns.
Do not accept a deceptive name such as googlebot.com.example.net. Check the actual DNS name from the resolver, not merely whether the text contains the word “googlebot.” Select the category that matches the request: a Google Ads fetcher is not automatically Googlebot, even when both are operated by Google.
3. Forward-resolve the returned name
Query the hostname you received:
host crawl-66-249-66-1.googlebot.comThe answer must include the original address, 66.249.66.1. Google’s example demonstrates this round trip. If the hostname does not resolve, resolves only to a different address, or belongs to an unexpected domain, do not treat the request as verified.
4. Record the result and apply policy
Keep the original IP, both DNS answers, resolver time and crawler category in your incident or change record. DNS answers can change, so a later investigation should repeat the lookup rather than rely on a screenshot or an old note. A verified identity still does not require you to permit unlimited crawling; robots rules, rate limits and application controls remain separate decisions.
Automated verification with Google’s published ranges
For firewalls, rate-limiters and high-volume log processing, match each source address against Google’s current CIDR objects. Google publishes separate resources for common crawlers, special-case crawlers and user-triggered fetchers. Use the list that corresponds to the caller you intend to recognize and refresh it instead of embedding a dated copy.
Why CIDR matching is preferable at scale
- It is deterministic and fast after the list is loaded.
- It avoids doing two DNS queries for every request.
- It lets you audit exactly which published network object matched.
- It can cover IPv4 and IPv6 when the relevant list contains both.
The live lists are maintained by Google and may change. The exact current contents are deliberately not reproduced here. See Verify Requests from Google Crawlers and Fetchers, then follow the links there to the appropriate range object. Google’s page was updated 2026-03-20 UTC; schedule refreshes and retain the retrieval time with each version.
Implementing a range check
The following Python example expects a local text file containing one CIDR per line, with blank lines and comments ignored. Obtain that file from Google’s category-specific published object, not from an unverified blog post.
from ipaddress import ip_address, ip_network
def load_networks(path):
networks = []
with open(path, encoding="utf-8") as f:
for raw in f:
line = raw.split("#", 1)[0].strip()
if line:
networks.append(ip_network(line, strict=False))
return networks
def in_google_ranges(address, networks):
ip = ip_address(address)
return any(ip in network for network in networks)
ranges = load_networks("google-common-crawlers.txt")
source_ip = "66.249.66.1"
print(in_google_ranges(source_ip, ranges))In production, load a new file atomically, validate that every entry parses as IPv4 or IPv6 CIDR, then swap the in-memory set. Keep the previous known-good list available so a malformed download cannot empty your allowlist. Log the matching network and list version for each enforcement decision.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose the right Google list
Use Google’s common crawlers for ordinary Googlebot-style crawling. If your log indicates AdsBot or another special-purpose agent, follow the special-case list and hostname rules in Google’s verification documentation. User-triggered fetchers are a separate category. A match in one list should not be silently generalized to every Google service.
Where ASN lookup fits
An Autonomous System Number identifies a network operator’s routing domain. Looking up the ASN for an IP can reveal that an address is announced by Google or a Google-associated network, which is useful context during triage. It is not the authentication procedure Google documents, and an ASN match alone does not establish that the request was generated by Googlebot.
Use ASN data to prioritize investigation or detect an address that is clearly outside the expected operator. For an allow or block decision, require the reverse/forward DNS round trip or a match in the correct, current Google CIDR object. Keep the ASN provider, lookup time and result if you use it, because commercial databases can differ and routing ownership can change.
Rank #4
DNS verification versus range matching
| Situation | Preferred method | Operational consideration |
|---|---|---|
| One or a few suspicious requests | Reverse DNS, domain check, then forward DNS | Interactive and directly documented; preserve both answers. |
| Many requests or an automated filter | Match the source IP against the appropriate published CIDR object | Refresh the list, validate downloads and select the correct crawler category. |
| ASN investigation | Supporting context | Do not use ownership alone as proof of Googlebot identity. |
Important edge cases
Google crawls from many locations
Googlebot can operate outside the United States. A country-based deny rule can therefore reject a genuine, verified request. Prefer identity checks based on DNS or published ranges rather than a presumed source country. Google’s locale-adaptive crawling guidance discusses this geographic distribution: How Google Crawls Locale-Adaptive Pages.
Free tools Windows power users keep installed
One-click scans. No signup required.
IPv6 and formatting
Normalize IPv6 before comparison. A textual comparison can fail when equivalent addresses use different compression or capitalization. Use an IP-library parser, as in the Python example, and preserve the original log value for auditability.
Proxies, CDNs and NAT
If your origin sees a CDN’s address, you cannot verify the end crawler from the origin log alone. Inspect the trusted edge log or configure the edge to pass a validated client address. NAT and shared infrastructure also mean that a network owner is not the same thing as a specific application.
DNS failures and transient answers
A timeout or temporary resolver failure is an “unverified” result, not proof of spoofing. Retry with an approved resolver, apply a bounded timeout, and avoid blocking an address permanently because of one failed lookup. Conversely, never bypass the forward check just because the reverse name looks convincing.
Troubleshooting checklist
- User-agent says Googlebot, DNS does not: treat it as spoofed or unverified; investigate the source IP and proxy path.
- Reverse name looks right but forward lookup differs: reject verification. The round trip failed.
- IP is in a Google-owned ASN but not your selected range: ASN is only context; verify the caller category and current list.
- Range file download changed unexpectedly: validate syntax, retain the prior list, and review Google’s category documentation before activation.
- Legitimate crawler blocked by geography: remove country-only assumptions and use the published ranges.
- IPv6 never matches: parse addresses as IP objects rather than comparing strings.
- Origin logs show only a proxy: perform verification at the trusted edge where the client address is known.
Or skip the browser setup
ScreenshotNeo is a separate option when your goal is to capture a page for investigation or monitoring rather than identify a crawler. It accepts a URL through one API call and can return PNG, JPEG, WebP or PDF. Before capture it accepts cookie/consent banners and removes more than 60 known consent platforms, newsletter popups and chat widgets; each step can be disabled. Bot checks, blank pages, timeouts, failed loads and cache hits are not billed, and the response reports the page verdict and billing status in X-Page-Verdict and X-Billed headers. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
See the ScreenshotNeo API documentation for authentication and options. cURL:
Best Value
- Used Book in Good Condition
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);Every plan includes its features: full-page and selector capture, device presets, retina scale, PDF controls, custom CSS/JavaScript, waits, request blocking, headers, cookies, user agents, timezone and geolocation, resizing, chosen-TTL caching, signed links, async webhooks, bulk capture of up to 100 URLs per call, usage API and OpenAPI support. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. Learn about ScreenshotNeo or sign up free.
Operational policy
Separate “verified Google infrastructure” from “allowed to fetch this endpoint.” Continue to enforce robots directives, authentication, rate limits and abuse controls after identity verification. Alert on repeated user-agent/IP mismatches, failed DNS round trips and range-list drift, and review exceptions when Google changes its published crawler categories.
Frequently Asked Questions
Does a Google ASN prove an IP is Googlebot?
No. ASN ownership is supporting context. Use the reverse-DNS/forward-DNS confirmation or the correct current Google CIDR list.
Should I hard-code the example 66.249.66.1 address?
No. Google’s example illustrates the procedure, not a complete or permanent allowlist.
Which address should I verify behind a CDN?
Verify the client address recorded by your trusted edge or proxy; the origin may see only the CDN connection.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
