Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Cloudflare CEO Matthew Prince said the company had blocked 416 billion requests from AI bots for its customers since July 1, 2025. The figure was reported in early December, roughly five months after that start date—not a full six months. It is a request-volume claim, not a count of proven attacks, unique pages, or documents taken.
What Cloudflare’s 416-billion figure means
The figure is attributed to Prince and Cloudflare; it is not an independently audited total for the entire internet. Search Engine Land’s report describes the claim as covering requests blocked since July 1. The phrase “in the last six months,” used in some headlines, rounds the period: in early December, July 1 was about five months earlier.
A request is an HTTP interaction with a website or resource. The total does not tell readers how many distinct pages or sites were involved, how many crawler operators made the requests, how much data was transferred, or how many requests resulted in a successful download. Repeated visits, retries, and requests across many Cloudflare-protected sites can all contribute to a request count.
Recommended Free Tools
Nor does “blocked” mean Cloudflare identified every request as malware or an illegal act. It means a Cloudflare control prevented the request from proceeding normally under the applicable policy or customer rule. The number alone does not establish copyright infringement, intent, or damages.
#1 Best Overall
- Compact and Efficient Design: The FortiGate 40F is designed for small to mid-sized businesses and enterprise branch offices, featuring a compact, fanless desktop form factor that ensures quiet operation and minimizes space usage.
- Robust Connectivity Options: Equipped with 5 GE RJ45 ports, including 1 WAN port and 4 internal ports, this model provides essential connectivity and flexibility for various network configurations in a small-scale environment.
- High-Performance Security: Offers up to 1 Gbps IPS throughput and 600 Mbps threat protection throughput, using Fortinet’s purpose-built security processor technology to deliver industry-leading performance and protection for SSL encrypted traffic.
- Advanced Threat Protection: Integrated with Fortinet’s AI-powered FortiGuard Labs, the FortiGate 40F offers comprehensive cybersecurity, identifying and mitigating both known and unknown threats to maintain robust security across your network.
- Simplified Management and Deployment: Features a user-friendly management console that provides comprehensive network automation and visibility, coupled with Zero Touch Integration with Fortinet’s Security Fabric for easy deployment.
Why publishers are reconsidering AI crawling
Traditional search crawlers gather pages for an index, and search engines can send visitors back through links. AI systems may collect content for model training, retrieve it to answer questions, or use it in other products. Those purposes are not interchangeable, and not every crawler associated with AI has the same value—or the same operator or policy.
Cloudflare has argued that some AI crawling provides publishers far less referral traffic than conventional search. In a June 2025 analysis, the company reported crawl-to-referral ratios of about 1,700:1 for OpenAI and 73,000:1 for Anthropic. Those are Cloudflare’s measurements, not universal ratios for every site or crawler. They nonetheless help explain why publishers may see automated access as a licensing and infrastructure question rather than an automatic route to audience growth. Cloudflare’s explanation of its AI-training controls describes that concern.
Rank #2
- 【Professional Firewall & NAS SERVER】OAKNODE 10gbe Firewall Appliance Mini PC-MGNASN, a powerful professional firewall router pc equipped with a 12th Gen Alder Lake N100 4C/4T up to 3.4GHz TDP only 6W with Intel UHD Graphics which maximizes the performance of the 2.5GbE port & SFP+ port, bring you a smooth secured and encrypted network environment.
- 【Rich I/O to meet your needs】Firewall Appliance MGNASN With HDMI 2.0+DP 1.4+TYPE-C(dp 1.2) Support for 3x4K@60Hz together, Dual DDR4 RAM slot support for up to 1x32GB SO-Dimm laptop DDR5 Ram Maximum 5600Mhz and 1xM.2 NVMe/PCIe 3.0x1 2280 SSD slot +1*SATA 3.0 SSD/HDD slots (install externally), also it support boot from TF card slot and it also support PXE/AWOL/Watchdog/GPIO etc. which is perfect for your firewall appliance、VM、Router、home Server needs.
- 【2xSFP+ 10GbE + 4x2.5GbE】This Firewall Router equipped with 2xIntel 82599ES 10gbe network card and 4*Intel i226-V network card speed maximum up to 2.5GbE(need other device like router, cables etc. also support 2.5Gbe/10gbe)which can bring you more faster and professional network usage(some system not release drivers yet) suggest to install version of below systems: pf-sense plus 23.0X or CE 2.7.X, OPNsense 22.1, OpenWrt, ROS7, ESXI 8 , Proxmox, CentOS etc).
- 【4G LTE Function supported】This model also support 4G LTE function(mini PCIE slot for 4G modem) and SIM card slot which you can use it as a IOT devices for your server.
- 【Quality With Warranty】If you have any questions or requirements(like OS installation/ drives/bios updates etc.) on OAKNODE Firewall mini pc MGNASN, PLEASE feel free to contact us. We offered 12 Months warranty for it and WE'LL REPLY YOUR Questions within 12 hours(during Workdays).
It is useful to distinguish at least four categories: training crawlers collecting material for model development; AI-search crawlers fetching pages for answers or citations; agents browsing on a user’s behalf; and malicious automation such as credential attacks or abusive scraping. A site may choose to block training crawlers while allowing a crawler that supports useful citations, or it may make a different choice based on its content and audience.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →What changed on July 1, 2025
On July 1, Cloudflare announced “Content Independence Day,” a policy and product push to give publishers more control over AI access, including the option to seek permission or compensation. The company introduced managed robots.txt controls for AI-training preferences, tools to block AI bots on parts of a site, and Pay Per Crawl in private beta. Cloudflare’s announcement sets out the policy direction.
Rank #3
- BUSINESS READY - pfSense+ software updates included for product lifetime. Netgate TAC Lite technical support included. One year hardware warranty included.
- COMPLETE - Pre-loaded with pfSense+ software to get up and running fast. Simply unbox it and start customizing for your secure edge networking needs. Free help with setup from our expert Technical Assistance Center (TAC) available 24/7/365.
- POWERFUL - A dual core ARM Cortex-A53 1.2 GHz delivers near gigabit routing of common home iPerf3 traffic and in excess of 650 Mbps of firewall throughput.
- COMPACT - Low power draw, a compact form factor, and silent operation allow it to run unnoticed when placed on a desktop, wall, or rack.
- FLEXIBLE - Three (3) 1 GbE switched (WAN/LAN/OPT) ports allow you to configure three separate 1 GbE switched ports for upto a gigabit of bi-directional traffic.
A managed robots.txt directive communicates a site owner’s preference to crawlers that comply with it. It is not an access lock: a crawler can ignore the file. For enforcement, a site needs controls such as Cloudflare rules, bot-management tools, an application firewall, or rules at its own server or network. The two approaches complement one another: one publishes the policy, while the other can deny requests.
What site owners can do with AI Crawl Control
Cloudflare’s current documentation describes AI Crawl Control as offering monitoring and basic controls across plans. Site owners can see AI crawler activity, allow or block individual crawlers, and track whether crawlers comply with robots.txt. Cloudflare says the product was formerly known as AI Audit. The available controls and dashboard labels can change; consult the current AI Crawl Control documentation for account-specific details.
Rank #4
- 【CPU】Intel Pentium J3710 4-Core/4-Thread processor, up to 2.64GHz, with 2MB L2 Cache and 6W TDP. Supports AES-NI and suitable for firewall, router, VPN and other network applications.
- 【Ports & Expansions】Equipped with 4 x 2.5GbE Intel i226-v LAN ports. Includes 2 x USB3.0, 1 x HDMI. 1 x VGA ports.Supports optional Wi-Fi and 3G/4G module expansion, plus a VESA mounting kit.
- 【Fanless & Low-Power Design】6W fanless design with an aluminum alloy chassis for quiet, low-maintenance operation. Design for 24/7 continuous use and suitable for home networks, small office and network labs.
- 【RAM & Storage】Includes 8G DDR3 RAM and a 128GB mSATA SSD. Supports up to 8GB RAM and 512GB mSATA storage. HDD storage is not supported. Compact 5.27 x 4.98 x 1.43-inch design weighs only apporximately 500g.
- 【Warranty & Support】Tested with pfSense, OPNsense, Ubuntu and other popular open-sourse OS. Supports Proxmox VE for virtualization and home lab applications. Includes a 12-month hardware warranty and lifetime technical support. (Press "DEL" to the BIOS)
Identification has limits. Controls may use known crawler identities, user-agent information, bot signals, Web Bot Auth, or other classification mechanisms, but a user-agent string can be forged. Blocking only named crawlers may miss unidentified or impersonating traffic. Conversely, a name in a request does not by itself prove the request came from the operator associated with that name.
Free tools Windows power users keep installed
One-click scans. No signup required.
Pay Per Crawl: a possible access charge, not guaranteed revenue
Cloudflare’s Pay Per Crawl is documented as a closed beta, not a generally available revenue stream. It is intended to let a site owner set a price for successful content retrieval by participating AI crawlers. Cloudflare’s documentation describes payment intent, a response of HTTP 402 Payment Required when acceptable payment intent is not supplied, and a documented minimum price of $0.01 per crawl. Charges are tied to successful retrieval; error responses are not billed. See the Pay Per Crawl overview and FAQ.
Best Value
- 【CPU Optimized for Firewall Mini PCs】This firewall appliance is powered by Intel Quad-Core Celeron J1900, 64-bit, up to 2.0 GHz, supporting software-based encryption. Energy-efficient and reliable, it runs 24/7 for home or small office networks, handling VPNs, multi-WAN routing, and basic firewall tasks efficiently.
- 【4×Intel i210 Ports】Equipped with four Intel i210 network controllers, each delivering up to 1 GbE for reliable multi-WAN routing, VPN connections, VLAN management, and stable performance in small office or home firewall deployments
- 【Memory & Storage】This Firewall Mini PC comes with 4 GB DDR3L RAM and a 64 GB mSATA SSD, providing reliable performance for basic networking tasks. AMI BIOS with ACPI support ensures stable system operation and energy-efficient 24/7 use
- 【Flexible System Compatibility】Compatible with Windows 10, Linux, and professional firewall systems such as pfSense, OPNsense, and VyOS, ensuring stable network management for home or small office use
- 【After-Sales Support:】This compact, fanless, and silent firewall keeps your network secure. Includes lifetime technical support and a 30-day money-back guarantee!
That price is a publisher-set per-crawl amount, not a Cloudflare subscription price or a promise of income. A crawler operator must support the payment flow and choose to pay. Re-crawls can generate additional charges, so participating operators need spending controls. Publishers should also consider whether one price makes sense for everything from archive pages to original reporting or documentation.
There is an important configuration trap: Cloudflare says WAF or Bot Management rules that already block a crawler take precedence over Pay Per Crawl. If a publisher wants the payment flow to run, it cannot simultaneously have a conflicting rule deny that crawler first. Cloudflare’s integration guidance explains this interaction.
Choosing whether to allow, monitor, block, or charge
| Goal | Possible approach | Trade-off |
|---|---|---|
| Preserve useful discovery and citations | Allow selected search or answer crawlers; monitor their activity. | Access may still create costs or use content without referrals the publisher considers sufficient. |
| Opt out of AI-training use | Publish managed or site-maintained robots.txt preferences and enforce them with access rules where needed. |
Compliant crawlers may honor the signal, but robots.txt alone does not stop a non-compliant crawler. |
| Reduce unwanted traffic or origin load | Block or rate-limit selected crawlers, including on specific paths. | A broad block can reduce visibility in AI search, citations, or services a site values. |
| Explore paid access | Consider Pay Per Crawl if eligible, and test pricing and rules carefully. | The feature is beta-gated and depends on crawler participation; blocking rules can prevent payment requests from reaching the flow. |
| Protect sensitive or commercially distinct paths | Use path-specific rules or other access controls rather than a site-wide decision. | More precise policies require ongoing configuration and monitoring. |
For many publishers, the practical choice is not simply “allow AI” or “block AI.” Training, search, and user-triggered access may deserve separate policies. A site should weigh the value of visibility and referrals against content sensitivity, bandwidth and origin costs, its licensing strategy, and its ability to enforce rules. Cloudflare’s controls can help manage that decision, but they do not settle questions of copyright, licensing scope, or whether a particular use is legally authorised.
What the headline cannot tell you
Without a breakdown, 416 billion requests cannot be converted responsibly into a count of articles, terabytes, unique visitors, financial losses, or infringement events. It also cannot show how many sites were affected or how many requests came from each crawler. The most defensible reading is narrower: Cloudflare says its controls blocked a very large volume of AI-associated requests for customers after its July 2025 policy shift.
The broader change is that publishers are moving from passive acceptance of crawling toward explicit choices about access: allow selected bots, monitor them, block them, or experiment with charging. The right choice varies by site, content, and crawler. The 416-billion figure illustrates the scale Cloudflare says it handled; it does not make that decision for publishers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
