Several Cloudflare controls can block or challenge AI agents, and some of them stop a request before the visitor ever sees a normal browser page. Cloudflare’s documentation does not show that any particular publisher turned away paying agents, and it does not quantify lost sales or revenue. Treat the seven settings below as failure modes to check on your own zone, not as a pattern confirmed across sites.
Start with Cloudflare’s three crawler categories
Cloudflare classifies bot traffic by behavior, not by brand. Its bot classification defines the Agent category as “automated activity acting in real time on a person’s behalf to get something done, such as chat fetch bots and browser-use agents.” Search crawlers and Training crawlers are separate categories.
As an Amazon Associate I earn from qualifying purchases.
That separation has practical consequences. Blocking Training traffic does not by itself block real-time agent retrieval, and allowing Search does not allow every assistant agent. A policy written for one category leaves the other two to whatever other rules are in place.
Free tools Windows power users keep installed
One-click scans. No signup required.
Seven settings that can turn away an agent
Each item below is a configuration path that Cloudflare documents. The list describes what each setting can do; it does not establish that all seven were active on any one site.
#1 Best Overall
- Compact and Efficient Design: The FortiGate 40F is designed for small to mid-sized businesses and enterprise branch offices, featuring a compact, fanless desktop form factor that ensures quiet operation and minimizes space usage.
- Robust Connectivity Options: Equipped with 5 GE RJ45 ports, including 1 WAN port and 4 internal ports, this model provides essential connectivity and flexibility for various network configurations in a small-scale environment.
- High-Performance Security: Offers up to 1 Gbps IPS throughput and 600 Mbps threat protection throughput, using Fortinet’s purpose-built security processor technology to deliver industry-leading performance and protection for SSL encrypted traffic.
- Advanced Threat Protection: Integrated with Fortinet’s AI-powered FortiGuard Labs, the FortiGate 40F offers comprehensive cybersecurity, identifying and mitigating both known and unknown threats to maintain robust security across your network.
- Simplified Management and Deployment: Features a user-friendly management console that provides comprehensive network automation and visibility, coupled with Zero Touch Integration with Fortinet’s Security Fabric for easy deployment.
1. AI Agent policy
This is the behavior-specific control for Agent traffic. Its options are Allow, Block on all pages, and Block on pages with ads. For real-time fetch and browser-use agents, it is the most direct policy path. Cloudflare’s Block AI Bots documentation, last updated July 1, 2026, lists September 15, 2026 as the effective date for new-domain defaults that block Agent bots on pages with ads. That date has passed, so a domain created since then may carry the block without anyone having chosen it. Check the zone’s saved value rather than assuming the default.
2. Legacy Block AI Bots
Cloudflare states that the legacy Block AI Bots switch is being deprecated on September 15, 2026. It targets AI training bots, and it treats mixed-purpose crawlers differently from the newer behavior-based policy. A legacy “AI bots” toggle should not be read as covering the Agent category. If it is still enabled, it can block requests that the Agent policy would have allowed, and the reverse is also possible.
Rank #2
- 【Professional Firewall & NAS SERVER】OAKNODE 10gbe Firewall Appliance Mini PC-MGNASN, a powerful professional firewall router pc equipped with a 12th Gen Alder Lake N100 4C/4T up to 3.4GHz TDP only 6W with Intel UHD Graphics which maximizes the performance of the 2.5GbE port & SFP+ port, bring you a smooth secured and encrypted network environment.
- 【Rich I/O to meet your needs】Firewall Appliance MGNASN With HDMI 2.0+DP 1.4+TYPE-C(dp 1.2) Support for 3x4K@60Hz together, Dual DDR4 RAM slot support for up to 1x32GB SO-Dimm laptop DDR5 Ram Maximum 5600Mhz and 1xM.2 NVMe/PCIe 3.0x1 2280 SSD slot +1*SATA 3.0 SSD/HDD slots (install externally), also it support boot from TF card slot and it also support PXE/AWOL/Watchdog/GPIO etc. which is perfect for your firewall appliance、VM、Router、home Server needs.
- 【2xSFP+ 10GbE + 4x2.5GbE】This Firewall Router equipped with 2xIntel 82599ES 10gbe network card and 4*Intel i226-V network card speed maximum up to 2.5GbE(need other device like router, cables etc. also support 2.5Gbe/10gbe)which can bring you more faster and professional network usage(some system not release drivers yet) suggest to install version of below systems: pf-sense plus 23.0X or CE 2.7.X, OPNsense 22.1, OpenWrt, ROS7, ESXI 8 , Proxmox, CentOS etc).
- 【4G LTE Function supported】This model also support 4G LTE function(mini PCIE slot for 4G modem) and SIM card slot which you can use it as a IOT devices for your server.
- 【Quality With Warranty】If you have any questions or requirements(like OS installation/ drives/bios updates etc.) on OAKNODE Firewall mini pc MGNASN, PLEASE feel free to contact us. We offered 12 Months warranty for it and WE'LL REPLY YOUR Questions within 12 hours(during Workdays).
3. Managed robots.txt
Cloudflare can prepend managed AI crawler disallow directives to an existing robots.txt file, or serve a managed file when the site has none. Cloudflare’s robots.txt setting documentation describes this as a crawler preference signal. If the owner wants enforcement, Cloudflare points to AI Crawl Control. When an agent is refused, first determine whether the client read a robots.txt directive and declined to proceed, or whether the edge returned a block or challenge. Those are different diagnoses with different fixes.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems4. Bot Fight Mode
Bot Fight Mode is the free product. It detects known bot patterns and issues computationally expensive challenges across the whole domain. Cloudflare warns that it may challenge API or mobile-app traffic. It runs in a separate evaluation pipeline, so WAF custom rules and Page Rules cannot bypass it. A non-browser client can fail on this setting even when no rule names it, which makes the failure hard to trace from the rule list alone. See Cloudflare’s Bot Fight Mode guide, last updated August 3, 2026.
Rank #3
- BUSINESS READY - pfSense+ software updates included for product lifetime. Netgate TAC Lite technical support included. One year hardware warranty included.
- COMPLETE - Pre-loaded with pfSense+ software to get up and running fast. Simply unbox it and start customizing for your secure edge networking needs. Free help with setup from our expert Technical Assistance Center (TAC) available 24/7/365.
- POWERFUL - A dual core ARM Cortex-A53 1.2 GHz delivers near gigabit routing of common home iPerf3 traffic and in excess of 650 Mbps of firewall throughput.
- COMPACT - Low power draw, a compact form factor, and silent operation allow it to run unnoticed when placed on a desktop, wall, or rack.
- FLEXIBLE - Three (3) 1 GbE switched (WAN/LAN/OPT) ports allow you to configure three separate 1 GbE switched ports for upto a gigabit of bi-directional traffic.
5. Super Bot Fight Mode
Super Bot Fight Mode is included in Pro, Business, and Enterprise plans. It adds configurable actions by bot category, analytics, static-resource protection, and WAF custom-rule exceptions. Cloudflare still warns that it may challenge API or mobile-app traffic. Check which categories are set to block or challenge. The exceptions mechanism is the practical difference from Bot Fight Mode, because a Super Bot Fight Mode configuration can be narrowed with custom rules while Bot Fight Mode cannot. See the Super Bot Fight Mode guide, last updated August 3, 2026.
6. WAF custom rules
Custom rules can block or challenge requests based on conditions you define. They execute before Super Bot Fight Mode managed rules, so a broad rule can stop a request before a later allow or monetization stage sees it. Custom rules also need manual upkeep, because the signals they match, such as headers and address ranges, change as agents change. Cloudflare’s custom rules documentation, last updated April 15, 2026, covers rule updates and execution order.
Rank #4
- 【Processor & OS】Firewall Mini PC with Intel J3710 CPU up to 2.64GHz, 4Cores 4threads 2MB L2 Cache, TDP 6.5w, supports AES-NI. It tested with pf-sens/opn-sense linux ubuntu and other popular open source os. ("DEL" key to enter BIOS)
- 【Interfaces】The firewall pc has 4 * Intel I226 lan ports, 2 * USB3.0 ports, 1 * RS232COM port, 2 * HD port, 1 * DC port. Equipped with VESA mount, you can install the micro pc behind the monitor to save space.
- 【Fanless Design】only 6.5W; fanless heat dissipation design, aluminum alloy shell, efficient and fast heat dissipation, which can withstand temperatures up to 60°C. support 24/7 hours working, no noise.
- 【RAM & Storage】The firewall router equipped with 8G DDR3 RAM, max support 8GB; 128GB mSATA SSD, up to 512GB. Not support HDD. Size:5.27 * 4.98 * 1.43 inches, Weigh:500g, small but powerful.
- 【12 Months Service】You will get a firewall pc and accessories，If you encounter any problems during the use, please contact us through Amazon, we have a professional and efficient team dedicated to serving you.
7. User-Agent blocking
User-Agent rules can block or challenge specified User-Agent headers across a domain. Cloudflare recommends custom rules when the goal is to block specific user agents. This mechanism is literal: it matches the string an agent sends, so it becomes stale when an agent changes its identifier. It is also a different diagnosis from Cloudflare’s verified bot classification, which is based on behavior. Details are in the User Agent Blocking documentation, last updated August 14, 2026.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Side-by-side view of the seven controls
| Setting | Effect | Scope | Position relative to other layers |
|---|---|---|---|
| AI Agent policy | Allow, block on all pages, or block on pages with ads | Agent behavior category | Upstream of pay per crawl |
| Legacy Block AI Bots | Block | AI training bots; mixed-purpose crawlers treated differently from the Agent policy | Upstream of pay per crawl; deprecation on September 15, 2026 |
| Managed robots.txt | Crawl preference signal, not enforcement | AI crawler disallow directives in robots.txt | Not an enforcement layer |
| Bot Fight Mode | Challenge | Whole domain, known bot patterns | Separate evaluation pipeline; not bypassed by WAF custom rules or Page Rules |
| Super Bot Fight Mode | Allow, challenge, or block by category | Bot categories; Pro, Business, and Enterprise plans | Bot solutions stage; supports WAF custom-rule exceptions |
| WAF custom rules | Block or challenge | Request conditions you define | Executes before Super Bot Fight Mode managed rules |
| User-Agent blocking | Block or challenge | Specified User-Agent headers, domain-wide | Not stated in the source for ordering |
Why an allow in robots.txt does not guarantee access
The question many site owners ask is why an agent is refused even though robots.txt allows it. The answer is that these are separate layers. robots.txt is a file the client chooses to read and respect; it is not technical enforcement. Behavior policies, WAF rules, and bot challenges are enforced at the edge, independently of what robots.txt says. An allow directive at one layer does not mean every other control allows the request.
Best Value
- 【CPU Optimized for Firewall Mini PCs】This firewall appliance is powered by Intel Quad-Core Celeron J1900, 64-bit, up to 2.0 GHz, supporting software-based encryption. Energy-efficient and reliable, it runs 24/7 for home or small office networks, handling VPNs, multi-WAN routing, and basic firewall tasks efficiently.
- 【4×Intel i210 Ports】Equipped with four Intel i210 network controllers, each delivering up to 1 GbE for reliable multi-WAN routing, VPN connections, VLAN management, and stable performance in small office or home firewall deployments
- 【Memory & Storage】This Firewall Mini PC comes with 4 GB DDR3L RAM and a 64 GB mSATA SSD, providing reliable performance for basic networking tasks. AMI BIOS with ACPI support ensures stable system operation and energy-efficient 24/7 use
- 【Flexible System Compatibility】Compatible with Windows 10, Linux, and professional firewall systems such as pfSense, OPNsense, and VyOS, ensuring stable network management for home or small office use
- 【After-Sales Support:】This compact, fanless, and silent firewall keeps your network secure. Includes lifetime technical support and a 30-day money-back guarantee!
Rule order decides whether monetization ever sees the request
Cloudflare’s AI Crawl Control with Cloudflare Bots documentation, last updated July 1, 2026, sets out the order in which these layers run for crawlers:
- WAF custom rules that block crawlers run first.
- Cloudflare bot solutions, Bot Fight Mode or Super Bot Fight Mode, run next.
- Pay per crawl charging occurs after the bot solutions.
Cloudflare notes that operators who plan to charge crawlers may need to turn off an upstream Block AI Bots control or adjust WAF rules so the request reaches the charging stage. A request blocked at an earlier stage is not a monetized request. It is a request that never reached the charge.
Challenges interrupt automated clients, not only blocked ones
Cloudflare’s challenge documentation, last updated July 6, 2026, maps WAF custom rules and rate limiting, Bot Fight Mode and Super Bot Fight Mode, and Under Attack Mode to challenge experiences. Bot Fight Mode and Super Bot Fight Mode can return interstitial challenges, and Under Attack Mode uses Managed Challenge. These challenges are designed for visitors who can complete the challenge flow. An automated client may therefore never receive the page, even when no rule explicitly rejected it as an AI agent. Outcomes depend on the client and on the configuration. The documentation establishes the challenge mechanism, not how every agent behaves when it meets one.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePay per crawl: what is charged and what is free
Cloudflare’s AI Crawl Control page, last updated August 14, 2026, describes pay per crawl as a private beta feature. Verify its availability before relying on it. The Pay Per Crawl FAQ, last updated April 23, 2026, sets out the billing rules:
- A successful crawl of protected content incurs the site owner’s configured price.
- Re-fetches are charged again.
- One configured price applies to all crawlers set to Charge.
- These paths are always free:
/robots.txt,/sitemap.xml,/security.txt,/.well-known/security.txt, and/crawlers.json.
How to check whether your zone is turning agents away
- Record the exact User-Agent string and the response the agent received, including whether it was a challenge page, a block page, or a 403-style refusal. Compare that with what a normal browser receives.
- Check the saved value of the AI Agent policy. Do not rely on the new-domain default.
- Confirm whether the legacy Block AI Bots control is still enabled.
- Read the robots.txt file for managed directives and determine whether the agent is respecting them.
- List WAF custom rules in their execution order and look for any rule whose conditions match the agent’s User-Agent, path, or address range.
- Review Bot Fight Mode or Super Bot Fight Mode category actions and their analytics for the same requests.
- Review User-Agent blocking rules for literal matches on the agent’s identifier.
- If you charge crawlers, confirm the request reaches the pay-per-crawl stage rather than being stopped earlier.
What evidence a lost-sales claim would need
A specific claim that paying agents were turned away requires evidence from the publisher’s own systems. Useful evidence includes edge or server logs showing agent requests that received challenge or block responses, analytics that record the action taken on each request, and a comparison of agent requests before and after a rule change. Without those records, the most defensible statement is that a configuration could block an agent, and that the cause should be confirmed in your own logs.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
