Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesThe Googlebot User-Agent string can help you identify Google Search crawler requests in server logs, but it does not prove a request came from Google. Anyone can copy the header. To verify a suspected Googlebot request, check the logged source IP with reverse DNS and confirm the returned hostname resolves back to that same IP—or compare the IP with Google’s appropriate published crawler ranges.
What is the Googlebot User-Agent?
Googlebot is Google’s general name for two Google Search crawler types: Googlebot Smartphone and Googlebot Desktop. A request’s HTTP User-Agent header can indicate which type made it. Google says most sites are primarily indexed using their mobile version, so most Googlebot Search requests are from the Smartphone crawler.
The header is descriptive text sent by the client. It is useful for filtering and investigating logs, but it is not an identity credential: a non-Google bot can send the same string.
Googlebot User-Agent strings
Google’s published common-crawler reference gives these example HTTP headers:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
- Attention-grabbing design meets the latest evolution of the Google Pixel Camera on the new Google Pixel 11 Pro; Gemini Intelligence helps manage details so you can live in the moment[1]; and the phone is available in two sizes
- Unlocked Android phone gives you the flexibility to change carriers and choose your own data plan: Works with Google Fi, Verizon, T-Mobile, AT&T, and other major carriers[2]
- Stay informed without looking at your screen: When your phone is face down, Pixel HiLight gently alerts you with subtle glowing lights when your favorite contacts are calling or you’re talking with Gemini; exclusive to Google Pixel 11 Pro phones
- Magic Capture catches the moment as you live it: With just one tap, Pixel 11 Pro captures video and photos, and automatically edits, crops, and unblurs a curated collection, ready to share – and you get the memory of how it felt to be in the moment
- Two new cameras for more brilliant photos: A larger telephoto sensor captures 30% more light for clear, beautiful photos and videos, even in the dark[3]; Pixel’s longest zoom ever helps you capture details from impressive distances[4]
Googlebot Smartphone
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
Google also lists less common forms, including Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and Googlebot/2.1 (+http://www.google.com/bot.html). The Chrome/W.X.Y.Z portion is a placeholder for a browser version that changes over time as Google tracks Chromium releases. For log searches, match the stable Googlebot marker rather than pinning a filter to one Chrome version. See Google’s Googlebot documentation and common crawler reference for the current examples.
How to find Googlebot in server logs
Search the request’s User-Agent field for Googlebot, then retain the request’s source IP address and timestamp for verification. A basic text search can find candidate lines; the exact command depends on your log format and where the logs are stored. For example, with a plain-text access log:
Rank #2
- Google Pixel 10a is a durable, everyday phone with more[1]; snap brilliant photography on a simple, powerful camera, get 30+ hours out of a full charge[2], and do more with helpful AI like Gemini[3]
- Unlocked Android phone gives you the flexibility to change carriers and choose your own data plan; it works with Google Fi, Verizon, T-Mobile, AT&T, and other major carriers
- Pixel 10a is sleek and durable, with a super smooth finish, scratch-resistant Corning Gorilla Glass 7i display, and IP68 water and dust protection[4]
- The Actua display with 3,000-nit peak brightness shows up clear as day, even in direct sunlight[5]
- Plan, create, and get more done with help from Gemini, your built-in AI assistant[3]; have it screen spam calls while you focus[6]; chat with Gemini to brainstorm your meal plan[7], or bring your ideas to life with Nano Banana[8]
grep -i 'Googlebot' /var/log/nginx/access.log
This finds lines containing the marker, including spoofed requests. It does not validate the client. If the log format stores the User-Agent in a structured field, query that field instead of searching the whole line. Also check whether a reverse proxy or CDN records the visitor IP in a forwarded header: verify the actual client address as established by your trusted proxy configuration, not an untrusted header that a visitor can set.
Googlebot Smartphone and Desktop are distinguishable in their HTTP headers, but both use the same Googlebot product token in robots.txt. You cannot target the Smartphone crawler separately from Desktop with that file. Google’s Googlebot documentation explains the crawler types and their robots.txt behavior.
How to verify a request is really from Googlebot
Google recommends checking the source IP using reverse DNS, then confirming the resulting hostname with a forward lookup. This two-way check helps prevent accepting a forged hostname as proof.
- Record the source IP. Take it from the request log for the suspected crawler. If traffic passes through a proxy, use the client IP your trusted infrastructure has resolved.
- Run a reverse lookup. On systems with the
hostutility, runhost 66.249.66.1, replacing the example address with the logged IP. - Check the returned hostname. Google’s verification documentation describes common Googlebot names matching
crawl-***-***-***-***.googlebot.comorgeo-crawl-***-***-***-***.geo.googlebot.com. A hostname that merely contains the word Google is not enough. - Run a forward lookup. Query the hostname returned by the reverse lookup, for example
host crawl-66-249-66-1.googlebot.com. Confirm that the result includes the original source IP,66.249.66.1in this example. - Use Google’s maintained IP files for automation. For larger-scale checks, compare the IP against the published range file that matches the traffic type: common crawlers, special-case crawlers, or other fetchers. Do not treat one category’s list as a universal list for every Google request.
Google’s verification guide demonstrates a reverse lookup of 66.249.66.1 returning crawl-66-249-66-1.googlebot.com, followed by a forward lookup resolving back to 66.249.66.1. The example illustrates the method; use Google’s current verification guidance and crawler IP range files rather than copying a static IP list into your application. That page reports an update date of March 20, 2026.
Free tools Windows power users keep installed
One-click scans. No signup required.
Google identifies crawlers and fetchers through the User-Agent, source IP, and reverse DNS hostname. Its infrastructure includes common crawlers, special-case crawlers, and user-triggered fetchers, and not every Google request follows the same rules. First establish what kind of request you are investigating, then use the corresponding verification method described in Google’s crawler overview and verification guide.
Googlebot verification troubleshooting
- The User-Agent says Googlebot, but DNS verification fails. Treat the request as unverified. The header can be spoofed; do not allowlist it or attribute its activity to Google based on that text alone.
- Reverse DNS returns a hostname outside the documented Googlebot patterns. Do not assume it is Googlebot. Check the appropriate official verification instructions for the crawler or fetcher type you are investigating.
- The forward lookup does not return the original IP. The reverse-and-forward check has not confirmed the request. Do not treat the hostname as proof; consult the current Google guidance and, for automated checks, the relevant published range file.
- A known Google request is absent from a Googlebot list. The request may be a special-case crawler or another fetcher, rather than a common Googlebot crawler. Identify the traffic category before choosing a range file.
- Your log shows a proxy or load balancer address. Configure logging to capture the client IP using your infrastructure’s trusted-proxy rules. Do not blindly trust client-supplied forwarding headers.
Robots.txt controls crawling, not guaranteed search removal
Google’s common crawlers, including Googlebot, follow robots.txt for automatic crawling. A Googlebot rule applies to both Smartphone and Desktop; it cannot distinguish them.
Blocking a URL in robots.txt is not the same as preventing it from appearing in Search. Google explains that a URL may still appear even when Googlebot cannot crawl the page. If the goal is to keep a page out of the index, use an indexing directive such as noindex where Google can access and process it. If the goal is to prevent access by crawlers and people, use access control such as password protection. Choose the control based on whether you intend to restrict crawling, indexing, or access.
For the current behavior and distinctions, consult Google’s Googlebot guidance and crawler overview.
Rank #4
- Google Pixel 10 Pro is the ultimate Pixel experience, featuring advanced AI with Gemini, unbelievable camera quality, impeccable design in two sizes, and the next-gen Google Tensor G5 chip[1]
- Unlocked Android phone gives you the flexibility to change carriers and choose your own data plan[2]; it works - Google Fi, Verizon, T-Mobile, AT&T, and other major carriers
- Get a head start on syncing your data before it even arrives: After you purchase your new Pixel, look for an email that explains how to transfer your photos, videos, passwords, and more in just a few quick steps[11]
- Pixel’s pro camera system makes everything look amazing, even in low light; capture more of the scene with advanced Google AI models, and bring out incredible details with 100x Pro Res Zoom, stunning 50 MP images, and super steady videos in 8K[10]
- Pixel 10 Pro is built with durable aluminum and Corning Gorilla Glass Victus 2 for scratch and drop resistance; the 6.3-inch Super Actua display with 3,300-nit peak brightness is easy on the eyes, even in direct sunlight[3,13,18]
Important limits and locale behavior
Google Search Central states that Googlebot currently fetches up to 2 MB for an individual URL, excluding PDFs; the stated PDF limit is 64 MB. These are fetch limits, not typical page sizes or a guarantee that all fetched content will be indexed. Google’s documentation says processing considers only the downloaded portion once the limit is reached. See the March 2026 Search Central update and the Googlebot documentation.
For locale-adaptive pages, Google says it uses the same User-Agent across crawling configurations, including geo-distributed crawling; the crawler IP may be based outside the United States. Do not rely on inferred visitor location to deliver a critical language or regional version only to some crawlers. Google recommends separate locale URLs with hreflang annotations. See its multi-regional and multilingual site guidance.
Capture a page yourself—or use an API
If you need to inspect what a page looks like, distinguish that task from verifying Googlebot. A browser screenshot shows rendered content; it does not authenticate a request seen in your logs. For manual inspection, load the page in a browser and use its developer tools or screenshot function. For repeatable captures, an API can return an image or PDF without you managing a browser installation.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server from Yorker Media. Make one GET request with a URL to receive a PNG, JPEG, WebP, or PDF. The cURL example below saves a WebP screenshot of the target page:
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Best Value
- Google Pixel 7 is powered by Google Tensor G2; it’s faster, more efficient, and more secure, with the best photo and video quality yet on Pixel[1].Other camera description:Front,Rear.Bluetooth Version 5.2 with dual antennas for enhanced quality and connection.
- Unlocked Android 5G phone gives you the flexibility to change carriers and choose your own data plan[2]; works with Google Fi, Verizon, T-Mobile, AT&T, and other major carriers
- Pixel’s Adaptive Battery can last over 24 hours; when Extreme Battery Saver is turned on, it can last up to 72 hours[3]
- The 6.3-inch Pixel 7 display is super sharp, with rich, vivid colors; it’s fast and responsive for smoother gaming, scrolling, and moving between apps[4]
- Google Pixel 7 has wide and ultrawide lenses with up to 8x Super Res Zoom[5]; and Cinematic Blur brings more drama to your videos
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webp
See the ScreenshotNeo API documentation for request options, or visit ScreenshotNeo.
ScreenshotNeo can accept cookie or consent banners before capture and remove more than 60 known consent platforms, newsletter popups, and chat widgets; each of those steps can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response indicates the page verdict and billing status in headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 screenshots; yearly billing gives two months free. Every feature is available on every plan. Sign up for free: 1,000 screenshots a month, no card required.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchFrequently Asked Questions
Can I use robots.txt to block only Googlebot Smartphone?
No. Google’s Googlebot token in robots.txt applies to both Smartphone and Desktop crawlers.
Does a Googlebot-looking User-Agent prove the request came from Google?
No. It is self-reported text and can be copied. Verify the source IP using Google’s reverse-and-forward DNS method or the appropriate published range file.
Does blocking a URL in robots.txt guarantee it will not appear in Google Search?
No. A blocked URL may still appear in Search; crawling controls and indexing controls serve different purposes.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →
