To identify and control AI bots, first measure which requests are driving load, then distinguish a crawler’s claimed identity from a verified one, decide which kinds of automated access you want, and apply proportionate rules at your CDN or WAF. Use robots.txt to communicate with cooperative crawlers; use edge enforcement for traffic that ignores the policy or exceeds your site’s limits.
First determine what is excessive for your site
There is no universal requests-per-minute threshold that makes an AI crawler excessive. Define the problem by its effect on your site: origin load, bandwidth, error rates, costs, or contention with legitimate visitors. A steady crawl of public pages may be harmless, while bursts against search, API, or computationally expensive routes may be costly.
As an Amazon Associate I earn from qualifying purchases.
Use access logs or CDN/WAF analytics to group requests by time window, URL path, response status, claimed user agent, source address or network, and burst or concurrency pattern. Look for repeated retrievals of expensive pages, APIs, search results, and large assets. Cloudflare’s AI Crawl Control reports crawler request counts and trends as well as robots.txt violations; AWS WAF Bot Control can label detected traffic by bot category and name and expose labels in metrics and logs (Cloudflare AI Crawl Control; AWS WAF Bot Control).
Identify the crawler without treating its name as proof
A user-agent string is a request header supplied by the client, not authentication. Bots can spoof it, so use the string as a classification clue rather than proof of origin. Cross-check it against request patterns, source network or provider identity signals, and managed bot-detection or verified-bot facilities where your CDN/WAF offers them. AWS specifically warns that bots may spoof user-agent headers (AWS guidance on managing AI bots).
#1 Best Overall
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 2 x vCPU core
- Fortinet HW FWB-VM02
- Manufacturer Part: FWB-VM02
Do not assume every AI-related request represents the same purpose. Cloudflare’s reference distinguishes OpenAI’s GPTBot, OAI-SearchBot, and ChatGPT-User, and Anthropic’s ClaudeBot, Claude-SearchBot, and Claude-User. It also lists other identifiers, including PerplexityBot, Bytespider, CCBot, and Google-CloudVertexBot. Names and classifications can change, so check the current reference when building or reviewing rules (Cloudflare bot reference).
- Training crawlers: requests associated with collecting content for model training.
- Search crawlers: requests associated with indexing or search features.
- Assistant fetches: requests made when a user asks an assistant to retrieve or summarize a page.
These distinctions matter because a rule targeting one class may leave another untouched. Decide which access you want before using a broad label such as “AI bots.” For managed detection, capabilities vary: AWS Bot Control’s common inspection level labels self-identifying bots, while its targeted level adds browser interrogation, fingerprinting, behavioral heuristics, and optional machine-learning traffic analysis. Cloudflare Bot Management customers can use detection IDs in custom WAF rules; available signals and features depend on service and plan (AWS WAF Bot Control; Cloudflare bot reference).
Rank #2
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 4 x vCPU core
- Fortinet HW FWB-VM04
- Manufacturer Part: FWB-VM04
Choose what access to allow before blocking
Set a policy by crawler purpose and path, not just by vendor name. You might permit search indexing on public pages while disallowing training crawlers from selected content, or allow user-requested retrieval while limiting automated collection. Preserve ordinary search-engine access if that is part of your goal, and test the actual CDN/WAF behavior as well as the text in robots.txt.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Where a provider publishes separate identifiers, prefer a narrow rule for the class you intend to control. A broad user-agent or “AI crawler” rule can have a different effect from blocking only a training crawler, and may interfere with search or user-initiated assistant access that you intended to retain.
Rank #3
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 8 x vCPU core
- Fortinet HW FWB-VM08
- Manufacturer Part: FWB-VM08
Use robots.txt to communicate with cooperative crawlers
A robots.txt file can request that compliant crawlers avoid particular paths or the site. AWS’s examples show how an operator can permit AI search crawlers on /public/ while disallowing access to /private/. AWS also documents Google-Extended and Applebot-Extended directives for expressing model-training preferences while retaining search indexing in those specific cases. These are crawler-specific examples, not universal directives; check how each operator says it interprets them (AWS guidance on managing AI bots).
robots.txt is a policy request, not access control. Some operators may not honor it, and a client can claim a permitted user agent while being something else. Do not put secrets or private content behind a disallow rule: protect private resources with authentication and authorization, and use edge rules when you need to stop requests from reaching your origin.
Rank #4
- Meraki MX100: A building block for SASE in a rack-mountable form factor. Medium- to large-branch security and SD-WAN appliance for up to 500 users.
- WAN: 1 x GbE RJ45, 1 x USB (cellular failover), Dual-purpose: 1 x GbE RJ45 +++ LAN: 8 x GbE RJ45, 2 x GbE SFP
- Stateful firewall throughput: 750 Mbps +++ 500 Mbps site-to-site VPN throughput
- Unified management for security, SD-WAN, Wi-Fi, switching, MDM, and IoT +++ Centralized management via web-based dashboard or API
- True zero-touch provisioning +++ Smartphone-like firmware updates
Enforce limits at the CDN or WAF
A CDN/WAF can inspect requests at the edge and allow, block, rate-limit, or challenge traffic before it reaches the origin. Choose the least disruptive action that addresses the impact you measured.
- Allow: preserve access for a verified desirable crawler or a client that meets your policy.
- Rate-limit: restrict high request volumes, whether they come from a named bot or another source. Set limits from your own traffic and capacity data; the cited guidance does not establish a generally correct threshold.
- Challenge: make uncertain or evasive traffic prove it is likely to be a browser before continuing. Challenges can be a better fit than an outright block when false positives would disrupt users.
- Block: stop a known unwanted identity or traffic pattern when the rule is sufficiently reliable and the impact warrants it.
AWS WAF Bot Control labels detected requests by category and name so you can match those labels in custom rules. Its targeted inspection can challenge bots that do not self-identify. AWS separately recommends rate-based rules to protect against high-volume sources and challenges for evasive scrapers (AWS WAF Bot Control; AWS guidance on managing AI bots).
Best Value
- ◆Powerful Celeron N2840 Processor: N2840 Processor, 2 Cores 2 Threads, 1M Cache, Max Turbo Frequency 2.58 GHz, TDP 7.5 W. Whether you need a robust home server, a versatile tool for school education, seamless web browsing, or even efficient business office or industrial tasks, providing efficient performance for everyday tasks.
- ◆Dual 1000M LAN: Mini Router PC with 2*Realtek RTL8111H network card chip full UDE 1000M with filter connector.Soft Router can monitor network data, improve network security, powerful and widely used.
- ◆DDR3L Memory & Large Storage Capacity: Firewall box computer with 1 x DDR3L SO-DIMM memory 1333/1600MHz, 1xMSATA3.0 SSD.
- ◆UHD Graphics & 4K Dual Screen Display: N2840 processor integrated UHD Graphics, HD and VGA dual display interfaces support 4K@60Hz.
- ◆Versatile Connections ports: 2 x1000M Realtek RTL8111H-LAN,2 xUSB3.0, 4 xUSB2.0, HDMI,VGA,AUDIO supports data storage and system boot.Mini desktop computer with WIFI dual antenna, which providing high-speed transmission and reliable connectivity. Support Dual Band Wifi, Internet, streaming media and audio can be used perfectly without interrupting the connection. Enjoy faster file transfers and smoother online experiences.
Cloudflare documents managed controls for AI crawlers and managed robots.txt, along with custom rules that can target paths and combine fields such as URI path, country, ASN, fingerprint, and user agent. Feature availability varies by plan: its documentation lists some AI crawler controls for all plans, while bot scores, verified bots, and certain bot-management fields require specific plans or subscriptions. Cloudflare also states that custom rules run before Super Bot Fight Mode rules, so a terminating custom action can prevent later bot settings from running. Check the current feature requirements and rule order before deploying (Cloudflare AI Crawl Control; Cloudflare custom rules).
Roll out rules gradually and watch for collateral effects
- Observe first. Review analytics, logs, labels, and the paths receiving traffic before applying a broad block. Cloudflare recommends using Bot Analytics before rules and increasing thresholds gradually; AWS recommends reviewing labels and logs to understand Bot Control detections before switching to blocking (Cloudflare: Challenge bad bots; AWS guidance on managing AI bots).
- Start with a narrow scope. Target a specific high-volume path, a well-classified crawler, or both. Avoid combining a broad identity match with a site-wide block unless that is the intended policy.
- Choose a measured action. Use a rate limit or challenge for uncertain traffic; reserve blocking for traffic you have reason to reject.
- Compare the result. Check request counts, origin load, errors, and whether desired crawlers or authenticated clients were affected. Add exceptions where needed.
- Expand only when the rule works. Adjust scope and limits in response to observed traffic, then keep monitoring as crawler identifiers and behavior change.
Cloudflare Bot Control carries additional fees. Other feature and plan limits can also affect which signals or controls are available, so compare them against infrastructure you already operate, desired detection depth, path-level control, logging, challenge behavior, and rollback needs before choosing a service (AWS WAF Bot Control; Cloudflare custom rules).
Verify certain ChatGPT Work Cloud browser requests with signed identity
OpenAI documents Web Bot Auth for requests from the ChatGPT Work Cloud browser. Those requests carry HTTP Message Signatures and a Signature-Agent value; operators can validate them using OpenAI’s published public keys. OpenAI provides recognition or allowlisting instructions for Cloudflare, Akamai, and HUMAN. This method applies to the documented ChatGPT Work Cloud browser requests only; it is not a way to authenticate all AI crawlers or all traffic claiming to come from ChatGPT (OpenAI: ChatGPT Work’s Cloud browser allowlisting).
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
