Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesSearch your server or CDN access logs for a crawler’s documented User-Agent token, then verify important matches against that operator’s published IP ranges or DNS-check procedure. A User-Agent alone is only a claim about who sent the request; robots.txt is a control file, not a visit report.
Where to look for crawler requests
Use the access log at the layer that receives and records the request. If a CDN or reverse proxy serves a request without contacting your origin, the origin log may not contain it; check the CDN or proxy logs as well as origin logs when relevant.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Windows Server 2012 Automation with PowerShell Cookbook | $63.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
Log fields vary by server and provider. When available, retain the timestamp, source IP, request path, response status, and full User-Agent. Together, these details help you investigate when the request occurred, what it requested, and how your site responded.
How to find candidate AI crawler requests
- Identify the User-Agent field in the access-log format used by your server, CDN, or proxy.
- Filter that field case-insensitively for documented crawler names. For OpenAI, its documentation gives distinct examples for
GPTBotandOAI-SearchBot; consult the current OpenAI crawler documentation for names, example strings, and IP ranges. - Match the stable name rather than a complete, version-specific User-Agent string. Versions and published crawler details can change. Google likewise advises allowing for version variation when matching crawler User-Agents; see its common crawler documentation.
- Save matching entries with their timestamps, source IPs, paths, response statuses, and full User-Agents, if those fields are available.
- Classify a User-Agent match as a claimed crawler until you verify it. Keep verified, unverified, and unknown requests distinct in reports.
OpenAI lists GPTBot and OAI-SearchBot as separate identities, so do not merge them into one category. Check the operator’s current documentation for the role and policy associated with each name.
#1 Best Overall
How to verify that a crawler is genuine
A request can claim any User-Agent string. Verification is operator-specific: use the crawler operator’s own published IP information or documented DNS procedure rather than assuming one company’s checks apply to another.
Googlebot
For a request claiming to be Googlebot, Google recommends matching the source IP against its published crawler IP ranges or using reverse DNS followed by a forward lookup. With the DNS method, reverse-resolve the request IP, then confirm that the resulting hostname resolves forward to the original IP. Google explains this in its request verification guidance and its Googlebot documentation.
OpenAI crawlers
OpenAI publishes IP ranges for its documented crawlers. Compare a request’s source IP with the current ranges in the OpenAI crawler documentation; avoid relying on a copied range that may become stale.
Free tools Windows power users keep installed
One-click scans. No signup required.
Other operators
Use each operator’s current official guidance for its User-Agent, IP ranges, and verification method. Perplexity’s community announcement links to a crawler guide that covers those subjects, but the announcement itself does not establish a current token or IP inventory: Perplexity crawler-guide announcement. Do not infer an exact identity or verification endpoint from a name alone.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to interpret robots.txt correctly
Robots.txt expresses crawler access preferences; it does not record traffic. A rule there does not prove that a crawler visited, nor that it did not. Use access logs to find requests observed by the logging layer.
Also distinguish a robots.txt control token from a request identity. Google documents Google-Extended as a standalone product token used for crawler-use controls, not as a request crawler identity equivalent to Googlebot. Searching logs for every robots.txt token will therefore produce misleading expectations.
Keep crawler records current and scoped
- Recheck operator documentation when creating filters, allowlists, or recurring reports; names, User-Agent versions, IP ranges, and guidance can change.
- Do not treat an old IP list or an unsuccessful DNS check as conclusive proof of spoofing. Check the current operator instructions and investigate the request in context.
- Interpret findings only for the logging layer you examined. An absent entry means that layer did not record the request; it does not by itself establish what happened elsewhere in the delivery path.
- Do not infer indexing, model training, search visibility, or inclusion in an AI answer from a logged visit. A request record establishes only what that log captures.
For large or recurring log datasets, log-search and observability tools can make filtering and retention easier; the same distinction between a claimed User-Agent and a verified identity still applies.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
