What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To see which bots and AI crawlers visit a React site, read the request logs at the layer that receives your traffic: your host, origin server, or CDN. Filter the user-agent field for known crawler names, then check the requested path and HTTP status code for each hit. A React build does not keep a crawler log of its own, so the exact console you need depends on where the site is deployed.
Why the log has to come from the receiving layer
A React app is usually compiled into static files, such as an HTML shell, JavaScript bundles, CSS, and images, which a hosting provider, web server, or CDN delivers. Every request for those files is recorded where it lands, not inside the React code. Cloudflare’s guidance on detecting AI crawlers takes the same approach: search the logs by crawler user-agent string and use the matching records to see which pages are requested and how often. Retention periods and dashboard layouts differ by provider, so confirm them in your host’s documentation before you rely on a report.
As an Amazon Associate I earn from qualifying purchases.
Workflow for finding crawler requests
- Find the layer that serves your domain. If your DNS points through Cloudflare, use the request logs or analytics for that zone. If you use another host, open its request log or traffic console. Menu names change often, so follow the provider’s current log documentation rather than a screenshot from an older guide.
- Filter for known crawler tokens. Start with
GPTBot,OAI-SearchBot,ChatGPT-User,ClaudeBot,Claude-SearchBot,Claude-User,PerplexityBot, andGooglebot. Treat this as a starting list, not a complete one. New tokens appear over time. - Keep context with every hit. For each record, note the timestamp, requested path, status code, and full user-agent string together. A hit on
/robots.txtand a hit on a product page tell you different things. - Classify the purpose before drawing conclusions. OpenAI and Anthropic each publish separate bots for different purposes (see the table below). Do not describe every AI-related request as training activity.
- Check how confident you are in the identity. A user-agent match is a clue. Apply operator verification where it matters to you.
- Separate discovery from enforcement. A robots file communicates preferences; it does not block requests. If you need blocking, use rules at your host or CDN.
On a self-managed Nginx server using the default combined log format on Debian or Ubuntu, a case-insensitive search might look like this:
grep -i -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Googlebot" /var/log/nginx/access.logThe log path and format differ on other servers and on managed platforms, so adjust the file location and the status-code field position to match your setup.
#1 Best Overall
- Hardware Controller with Professional Network Management-Centralized management for up to 100 Omada devices including Omada access points, Omada Security Gateways and Jetstream switches.
- Premium Hardware Design-Industry-leading flexible Rackmount/Desktop design with a powerful chipset, durable metal casing, 2 fast ethernet ports and 1 USB 2.0 port for auto backup.
- Dual power selection-Support PoE (802.3af/802.3at) and micro USB for flexible installations.
- Easy Network Monitor & Maintenance-The easy-to-use dashboard makes it simple to see your real-time network status and improve network maintenance for peace of mind.
- Cloud Access with No License Fee-Enjoy cloud service with no license fee with the use of OC200. Remote Cloud access and Omada app brings centralized cloud management of the whole network from different sites—all controlled from a single interface anywhere, anytime.
Crawler names and what each one is for
The table below lists the names cited in Cloudflare’s bot reference (last updated 23 April 2026) and the purposes stated by the operators’ own documentation. Cloudflare describes its list as a selection, and it points readers to Cloudflare Radar for the up-to-date directory.
| Operator | User-agent token | Stated purpose |
|---|---|---|
| OpenAI | GPTBot |
Crawling for potential use in training OpenAI’s generative AI foundation models (OpenAI crawler documentation, crawled 7 October 2026) |
| OpenAI | OAI-SearchBot |
Crawling for ChatGPT search results |
| OpenAI | ChatGPT-User |
Page access initiated by a user in ChatGPT |
| Anthropic | ClaudeBot |
Not stated in this guide’s sources; see Anthropic’s help center page on crawling, dated 7 April 2026 |
| Anthropic | Claude-SearchBot |
Not stated in this guide’s sources; see Anthropic’s help center page on crawling, dated 7 April 2026 |
| Anthropic | Claude-User |
Not stated in this guide’s sources; see Anthropic’s help center page on crawling, dated 7 April 2026 |
| Perplexity | PerplexityBot |
AI search (Cloudflare bot reference) |
Googlebot |
Search crawling; Google documents subtypes that can be identified from the HTTP user-agent header |
Reading status codes correctly
A status code tells you what the logging layer returned, not whether a page was read by a crawler that used the content. Keep these distinctions in mind when you count hits:
Rank #2
- Automatic Router Rebooter / Reset - Stop manually restarting your router! Automate the process to ensure highly reliable internet connection uptime
- Constantly Monitors Router and/or Modem Internet Health. Keep Connect provides 24/7/365 protection to ensure that your smart home and connected devices are always online and available.
- Notifications - Free Texts or Emails from Keep Connect notifying you of detected eventsif you choose to enter your phone number/email. You may also choose No Notifications.
- Perfect for Smart Home Reliability - Schedule Periodic Resets to keep your connection fresh and fast.
- Premium Cloud Services App Available (iOS App Store and Google Play Store) - Our Premium Keep Connect Cloud Services platform allows using our Online/Mobile App to monitor many locations in one place as well. Cloud Services allows remote management of devices at all locations as well as heartbeat monitoring of your Keep Connects to notify you in the event of an ISP internet outage at one of your sites.
- 200 means the request received a successful response from the layer that logged it.
- 403, 429, or another error code means the request was refused, rate-limited, or failed. Count these separately from successful page fetches.
- A request that your CDN answered from cache may not appear in logs at your origin server. Check the logs at the outermost layer that handles the request.
How sure can you be that a bot is who it claims?
A user-agent string is easy to copy, so a matching name is evidence, not proof. Cloudflare notes that some bots may not send an identifying header at all and may require other signals to identify. OpenAI publishes IP ranges for its documented crawlers, so comparing a hit’s source address against those ranges is a stronger check for OpenAI traffic. Until you have run a check of that kind, describe the hit as user-agent-matched.
Free tools Windows power users keep installed
One-click scans. No signup required.
Google’s documentation says Googlebot subtypes can be identified from the HTTP user-agent header, and it separates crawling from indexing. A page that never appears in your logs has not necessarily been dropped from search results, and a logged crawl does not guarantee indexing.
Rank #3
- (10/100/1G) Gigabit Bypass network tap / sniffer equivalent to port mirror on a switch.
- The two monitor/sniff ports are isolated from the network being monitored.
- Automatic bypass of device on power fail.
- Power-over-Ethernet (POE) pass-through. Rated at .75A max at 57vdc
- 5v power through USB3 port or 5v wall transformer (or both). ~500ma consumption.
Discovery is not enforcement: robots.txt and access controls
A robots.txt file tells compliant crawlers which paths you prefer they skip. It does not stop a request from reaching your server. Cloudflare’s guidance states: “Robots.txt is not binding — following it is more of a courtesy than anything else.” Check the file that is actually served at your domain, since a React build can ship a placeholder or an outdated copy.
OpenAI’s crawler documentation describes its controls as independent of each other: “Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.” Anthropic says its crawler honors standard robots directives, which is a statement about Anthropic’s bots specifically, not about every crawler on the web. If you need technical blocking, Cloudflare documents separate enforcement controls, and most hosts offer equivalent rules at their own layer.
Rank #4
- NEVER MANUALLY REBOOT YOUR ROUTER AGAIN – The ConnectSense Rebooter Pro plugs between your modem or router and the wall outlet, automatically detecting lost internet connectivity across up to 5 network targets and power cycling your equipment instantly — keeping your home, office, or remote location always online 24/7.
- SCHEDULED & AUTOMATIC REBOOTS – Set up to 10 custom reboot schedules to proactively clear memory leaks, prevent slowdowns, and keep your connection fresh — even before problems occur. Perfect for smart homes, security cameras, smart locks, thermostats, and any device that depends on a stable internet connection.
- REMOTE CONTROL FROM ANYWHERE – Trigger a manual reboot anytime from the free ConnectSense app (iOS & Android) or directly from your home network. Whether you're traveling, at work, or managing a vacation rental or remote office, you stay in control of your network without needing to be on-site.
- AUTOMATIC POWER OUTAGE RECOVERY – When the power goes out, the Rebooter Pro automatically restores and reboots your networking equipment once power returns, eliminating downtime and the need for manual intervention. Ideal for unattended locations, rental properties, and small business networks.
- INTEGRATOR & PRO-GRADE FEATURES – The only router rebooter with a built-in local HTTPS API, giving IT professionals, smart home integrators, and power users advanced automation, monitoring, and remote management capabilities — no cloud subscription required for local control.
Optional managed view: Cloudflare AI Crawl Control
If your site already runs through Cloudflare, AI Crawl Control gives a managed view of crawler traffic. Its analytics show request volume, allowed requests, status-code distribution, popular paths, and operators, with filters for each. It also offers per-crawler controls. Referral analytics are available only on paid plans, according to Cloudflare’s documentation (last updated 23 April 2026). You do not need this tool to read your logs, and a site that is hosted elsewhere can get the same filtering from its own log tools.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11What logs cannot tell you
- A crawler visit is not evidence of training use. Purpose depends on which bot made the request and that operator’s current documentation.
- No hits do not prove that no AI system has accessed your content, because requests may be served from cache, routed through another provider, or made by a bot with an unlisted token.
- Log retention is set by your host or CDN and may be shorter than you need for trend analysis. Export the records you want to keep.
If you are not sure which layer serves your site, start by checking where your domain’s DNS points. That answer determines which logs to search.
Quick Recap
Best Value
![[Upgraded] AURSINC NanoVNA-H Vector Network Analyzer 9KHz -1.5GHz Latest HW V3.7 HF VHF UHF Antenna Analyzer, Measuring S Parameters, SWR, Phase, Delay, Smith Chart](https://m.media-amazon.com/images/I/41hseV6xSbL._SL160_.jpg)
- [UPGRADED NanoVNA-H] New HW Version V3.7. It is upgradeable as new firmware is developed. With MicroSD card port now can have the measurement data or the screenshots saved in the it at anytime. Added battery circuit management, more secure. Redesigned PCB, you can connect to mobile phone with Type C-Type C cable (original PCB needs OTG cable), see a clear HD image on your phone. Added a ABS case, which is protective and dust-proof. Disply: 2.8 inch TFT (320 x240).
- [IMPROVED FREQUENCY ALGORITHM] The improved frequency algorithm can use the odd harmonic extension of si5351 to support the measurement frequency up to 1.5GHz. The 9KHz-300MHz frequency range of the si5351 direct output provides better than 70dB dynamic, The extended 300M-900MHz band provides better than 60dB of dynamics, and the 900M-1.5GHz band is better than 40dB of dynamics.
- [MULTIPLE FUNCTIONS] The default firmware main function is used for antenna performance measurement. The TX/RX method can measure the complete S11 and S21 parameters. If you need to obtain S12 and S22, you need to manually replace the transceiver port wiring. The CH0 output level is increased to 0dBm when using the fundamental wave, resulting in more accurate reflection measurement.
- [SUPPORT ANDROID PHONE & PC SOFTSARE CONTROL] Designed a practical and simple control application on PC, you can download touchstone(SNP) files for radio design and simulation software. There is a PC interface that adds functionality and lets you work interactively on a bigger screen. Supports time domain analysis function (TDR). Compatible with most Android mobile phones, convenient for connecting to mobile phones. Support Windows Computer Control.
- [STRONG AND SECURE POWER SUPPLY] This VNA is battery powered or USB powered. Built in 650mAh battery, could work for 2 hours continuously. For longer measurement time, kindly connect an external power source. The product interface displays battery usage, providing a clear understanding of the power status.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
