Use robots.txt to tell cooperative crawlers what you prefer them to access; use Cloudflare bot protection or another server-side control when you need to challenge or block requests. They work at different layers and can be used together: the file communicates a preference, while enforcement acts on traffic that reaches your site.
What does robots.txt do?
robots.txt is a plain-text file published at your site’s top-level /robots.txt path. It communicates crawl instructions to bots that choose to honor them. The IETF standard, RFC 9309, is explicit: “These rules are not a form of access authorization.” A crawler can ignore the file, and a client can spoof its user-agent string.
As an Amazon Associate I earn from qualifying purchases.
That makes robots.txt useful for coordinating with compliant crawlers, not for protecting private pages or stopping unwanted traffic. Cloudflare also describes its managed file as a statement of preferences rather than a technical barrier.
What does Cloudflare bot protection do?
Cloudflare’s bot products analyze automated traffic and can mitigate requests. Its bot solutions overview lists Bot Fight Mode, Super Bot Fight Mode, and Bot Management for Enterprise. Cloudflare’s WAF can also use built-in bot settings alongside custom rules; see its custom rules documentation.
#1 Best Overall
Unlike a crawler instruction file, request-level controls can challenge or block traffic. The exact controls and their granularity depend on the Cloudflare product and plan, so check what is available in your account. You can also enforce access rules through authentication, application logic, or controls at your origin server.
Which option should you use?
| Your goal | Best starting point | Reason |
|---|---|---|
| Tell compliant crawlers which paths they may crawl | robots.txt |
It expresses preferences in a protocol crawlers are asked to honor. |
| Challenge or block unwanted requests | Cloudflare bot controls, WAF rules, authentication, or origin controls | These apply enforcement to requests rather than relying on voluntary compliance. |
| Communicate a preference and enforce it | Use both | The file tells cooperative crawlers what you want; enforcement addresses requests that do not comply. |
| Keep search crawling while limiting other AI-related activity | Review crawler identity and Cloudflare behavior controls | Cloudflare documents Search, Agent, and Training categories, but a crawler’s purposes can overlap. |
This distinction follows RFC 9309 and Cloudflare’s managed robots.txt guidance; it does not mean every site needs Cloudflare.
Rank #2
- Protects against known exploits, malware and malicious websites; detects unknown attacks; identify thousands of applications
How Cloudflare’s robots.txt and AI controls fit together
Cloudflare’s managed robots.txt can add directives for known AI crawlers. If your origin already serves a robots.txt file, Cloudflare says its managed content is prepended to the existing file. Check the final response at /robots.txt and your zone configuration to confirm the resulting directives match your intent. The Bot Management API documentation is also available for account-level configuration details.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteCloudflare treats managed robots.txt and AI Crawl Control as separate mechanisms: the file communicates preferences, while AI Crawl Control is an enforcement option for blocking access. Cloudflare documents that they can be used together.
Rank #3
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 1 x vCPU core
- Fortinet HW FWB-VM01
- Manufacturer Part: FWB-VM01
What to check before changing AI bot settings
Cloudflare groups AI-related activity into three categories in its bot documentation:
- Search: content collection or indexing to answer questions later.
- Agent: real-time automated activity on a person’s behalf.
- Training: collection for training or fine-tuning models.
Cloudflare says all customers can manage these behaviors. Its documentation dated July 1, 2026 described defaults taking effect for new domains on September 15, 2026: Training and Agent blocked on pages displaying ads, with Search allowed. Because that effective date has passed and settings can change, do not assume the default applies to your zone. Review the current settings and Cloudflare’s AI bot blocking guidance. Also consider that one crawler may serve more than one purpose, so a category label may not capture every use you care about.
Quick Recap
A practical setup checklist
- Decide whether you want to communicate or enforce. Use
robots.txtfor crawler preferences. For a requirement that requests be stopped, choose an edge, application, authentication, or origin control. - Write crawler rules for the behavior you want. Check the file served at your domain’s
/robots.txtpath, including any Cloudflare-managed directives and existing origin content. - Choose enforcement that matches your traffic and account. Review the Cloudflare bot, WAF, or AI Crawl Control options available to your product and plan; use a different server-side control if it better fits your needs.
- Verify the result. Inspect the served
robots.txtand your active zone rules. A directive in the file alone does not prove that a request is technically blocked.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
