Free tools Windows power users keep installed
One-click scans. No signup required.
Do not trust a crawler’s User-Agent string as proof of identity. Treat it as a claim, verify the request’s network source using the operator’s published method, then separately decide whether the bot’s behavior fits your site’s rules. For Google traffic, the documented manual check is reverse DNS followed by forward DNS confirmation; signed Web Bot Auth requests can add cryptographic proof when available, but Google says not every request is signed.
What to check before allowing a crawler
Bot identification has two separate questions: who sent this request? and is this traffic acceptable for my site? A valid identity check can answer the first, but it does not mean you must allow the crawler. Keep identity verification, behavior monitoring, and access policy as distinct decisions.
As an Amazon Associate I earn from qualifying purchases.
- Identity evidence: source IP, DNS confirmation, published IP ranges, or a validated cryptographic signature.
- Behavior evidence: requested paths, timestamps, response codes, request rate, and whether the bot follows your crawl preferences.
- Policy decision: allow, rate-limit, block, or investigate based on your site’s requirements.
Why a User-Agent is not enough
A User-Agent is a self-reported label. A request can claim to be Googlebot or another well-known crawler without coming from that operator. Google explicitly warns that its crawler User-Agent is often spoofed and recommends checking the request source rather than trusting the string alone (Google: Verify requests from Google crawlers and fetchers).
Keep the User-Agent in your logs because it helps describe what the requester claims to be. Do not use it by itself as an allowlist rule.
#1 Best Overall
- Compact and Efficient Design: The FortiGate 40F is designed for small to mid-sized businesses and enterprise branch offices, featuring a compact, fanless desktop form factor that ensures quiet operation and minimizes space usage.
- Robust Connectivity Options: Equipped with 5 GE RJ45 ports, including 1 WAN port and 4 internal ports, this model provides essential connectivity and flexibility for various network configurations in a small-scale environment.
- High-Performance Security: Offers up to 1 Gbps IPS throughput and 600 Mbps threat protection throughput, using Fortinet’s purpose-built security processor technology to deliver industry-leading performance and protection for SSL encrypted traffic.
- Advanced Threat Protection: Integrated with Fortinet’s AI-powered FortiGuard Labs, the FortiGate 40F offers comprehensive cybersecurity, identifying and mitigating both known and unknown threats to maintain robust security across your network.
- Simplified Management and Deployment: Features a user-friendly management console that provides comprehensive network automation and visibility, coupled with Zero Touch Integration with Fortinet’s Security Fabric for easy deployment.
How to verify a Google crawler manually
Use the source IP that actually connected to your trusted server or edge. Google’s manual verification method checks that the IP and hostname corroborate each other:
- Take the request’s source IP from your access logs.
- Perform a reverse DNS lookup on that IP to find its hostname.
- Check that the hostname ends in an accepted Google domain for the relevant fetcher class:
googlebot.com,google.com, orgoogleusercontent.com. - Perform a forward DNS lookup on that hostname.
- Confirm that the returned address includes the original source IP.
A reverse lookup alone is not sufficient: the forward lookup checks that the hostname points back to the address that made the request. If a CDN or reverse proxy sits in front of your origin, verify the visitor IP supplied by trusted infrastructure; do not rely on an arbitrary forwarded header. Google recommends using its published IP ranges for regular automated verification rather than relying on manual lookups for every request (Google verification guidance).
Choose the right Google IP list
Google publishes separate IP information for common crawlers, special-case crawlers, and user-triggered fetchers. Identify the request class before applying a list or assuming a robots.txt rule:
Recommended Free Tools
Rank #2
- HARDWARE PLUS SECURITY SERVICES: FortiGate-60F Firewall Appliance bundled with 1 year of FortiCare Premium and FortiGuard Unified Threat Protection.
- UNIFIED THREAT PROTECTION (UTP): Secures against advanced online threats with comprehensive web filtering and anti-botnet technologies.
- OPTIMIZED FOR MEDIUM-SIZED BUSINESSES: Tailored for businesses needing robust security without the infrastructure of larger enterprises.
- RELIABLE CUSTOMER SUPPORT: FortiCare Premium ensures high-quality support and service continuity.
- EFFECTIVE PROTECTION: Employs advanced filtering technologies to safeguard against sophisticated threats.
| Request class | Examples and robots.txt behavior | Verification reference |
|---|---|---|
| Common crawlers | Includes Googlebot. These crawlers always respect robots.txt during automatic crawling. | Google crawler verification and common crawler ranges |
| Special-case crawlers | Includes AdsBot. These crawlers may or may not respect robots.txt. | Google special-case crawler ranges |
| User-triggered fetchers | Fetches initiated by a user; these ignore robots.txt because a user initiated the fetch. | Google user-triggered fetcher ranges |
Use the corresponding published data for the class you are checking. A Google-branded User-Agent does not establish which class sent a request, and the classes do not share identical robots.txt behavior. Google explains the categories and links to their published lists in its verification documentation.
How signed Web Bot Auth fits in
Web Bot Auth uses cryptographic signatures to let participating bots prove a key-backed identity. For Google-Agent requests, Google documents a Signature-Agent header naming https://agent.bot.goog; verification depends on validating the signature headers against the published key set, not merely seeing that header (Google: Authenticate requests with Web Bot Auth (experimental)).
Google describes its Web Bot Auth deployment as experimental and says it does not sign every request of a particular agent. Keep IP-range and DNS verification available as a fallback; a missing signature does not, by itself, show that a request is malicious.
Rank #3
- 【Up to 1100 Mbps VPN Speed 】 Hardware-accelerated WireGuard and OpenVPN-DCO deliver up to 1100 Mbps VPN throughput, over 3× faster than Brume 2 for smooth remote access and file transfers.
- 【Three 2.5G Ports & Multi-WAN】Tri-port 2.5GbE design with flexible WAN LAN configuration supports multi-gigabit wired setups, dual-ISP Multi-WAN and failover to keep home and SOHO networks online.
- 【Stealth VPN Obfuscation】VPN obfuscation disguises VPN traffic as regular HTTPS, helping you evade blocking, bypass restrictive networks and maintain stable, private connections.
- 【DPI protection】Deep Packet Inspection with visual dashboards blocks adult/gambling/malicious sites, while SQM and QoS prioritize gaming, calls, and video when bandwidth is tight
- 【OpenWrt & USB 3.0 Expansion】OpenWrt with 1GB DDR4 and 8GB eMMC lets you install plugins and build VPN, ad-blocking or NAS, while USB 3.0 Type‑C connects high-speed storage or 4G/5G dongles
Assess behavior after identity
Authentication tells you something about who sent a request, not whether the traffic is wanted or well-behaved. Cloudflare’s verified-bot guidance considers both honest identification and non-abusive behavior, including following crawl directives, using reasonable request rates, and not evading owner preferences or attacking sites (Cloudflare: Verified bots).
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Cloudflare’s classifications include Search, Agent, Training, Data Collection, and Security Testing; a bot can have more than one behavior. These labels can help describe a bot’s purpose, but they do not replace your own access policy. See Cloudflare’s verified-bot documentation for its classification framework and controls.
Quick Recap
- Compare request volume and timing with the rate your service can support.
- Check whether requested paths and crawl behavior align with your published preferences.
- Apply your site policy to the relevant bot class, even when identity is verified.
- Rate-limit or block traffic that exceeds your limits or disregards your rules.
Pick a verification method for the job
| Method | What it establishes | Coverage and practical use |
|---|---|---|
| User-Agent only | What the requester claims to be; it is weak identity evidence because it can be spoofed. | Useful as a log label, not as a standalone allowlist check. |
| Reverse and forward DNS | For Google requests, checks that the source IP resolves to an accepted hostname and that hostname resolves back to the source IP. | Useful for investigating individual requests; Google also recommends published ranges for routine automated checks. |
| Published IP ranges | Whether the logged source IP matches the appropriate operator-published crawler or fetcher range. | Better suited to high-volume automation; use the right list for the request class. |
| Validated Web Bot Auth signature | Cryptographic identity for a participating signed request when the signature validates against the documented key set. | Stronger evidence for signed traffic, but Google’s experimental deployment does not sign every request. |
A practical workflow for bot traffic in your logs
- Preserve request evidence. Record source IP, User-Agent, path, timestamp, response code, and request rate. If traffic passes through a CDN or proxy, establish which trusted field contains the actual visitor IP.
- Classify the claim. Note the User-Agent and any signature-related headers, but do not authorize a bot based on a name or header alone.
- Verify using operator documentation. For Google, use the DNS procedure or the appropriate published IP list. For another AI crawler operator, consult that operator’s official identity and IP or signature documentation; there is no single established verification recipe for every operator.
- Evaluate behavior. Compare the traffic with your request-rate limits, crawl preferences, and site policy.
- Enforce the decision. Allow, rate-limit, block, or investigate according to identity evidence and observed behavior, and retain enough logs to reassess later.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
