Cloudflare says Perplexity continued obtaining content from test websites after those sites blocked Perplexity’s declared crawlers with robots.txt rules and web-application firewall (WAF) controls. Perplexity disputes that interpretation, arguing that some traffic was misattributed to an automated browser service and that its fetching was triggered by user requests. The published accounts establish a serious, technically specific dispute—not an independent finding that Perplexity routinely bypasses every website’s rules.
What did Cloudflare allege?
In an August 4, 2025 report, Cloudflare described tests involving newly purchased domains that, it said, were not indexed or otherwise publicly discoverable. Cloudflare placed disallow directives in each domain’s robots.txt file and added WAF rules blocking Perplexity’s publicly declared crawlers. Despite those measures, Cloudflare said Perplexity answered questions about content placed on the test sites.
As an Amazon Associate I earn from qualifying purchases.
Cloudflare characterized the result as evidence that requests continued through traffic it did not associate with Perplexity’s published crawler identity. Its post says the observed traffic included both Perplexity’s declared user agent and a generic browser string identifying itself as Chrome on macOS. Cloudflare also said the latter traffic came from IP addresses outside Perplexity’s published ranges and shifted among IP addresses and autonomous systems after blocks were applied.
Cloudflare said it used machine-learning and network signals to identify the traffic. Those methodology and attribution claims come from Cloudflare; the sources reviewed do not provide a neutral third-party reproduction or adjudication.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What did Cloudflare’s numbers represent?
| Figure | What Cloudflare said it measured |
|---|---|
| 20–25 million requests per day | Cloudflare’s 2025 estimate for requests using the declared Perplexity-User user agent. |
| 3–6 million requests per day | Cloudflare’s 2025 estimate for traffic using the Chrome-like user agent it labeled stealth traffic. |
| Tens of thousands of domains | The scale of domains across which Cloudflare said it observed the activity. |
| More than 2.5 million websites | Cloudflare’s reported count of sites using its managed robots.txt feature or managed AI-crawler blocking rule to disallow AI training when the post was published. |
These are Cloudflare-reported observations or product-adoption figures, not independently verified industry statistics. The request estimates should not be read as a measurement of all Perplexity traffic or proof that every fetch ignored a site’s preferences.
How did Perplexity respond?
Perplexity disputed Cloudflare’s account. In a response reproduced by Daring Fireball, the company said Cloudflare had either sought a publicity moment or misattributed 3–6 million daily requests from BrowserBase’s automated browser service. Search Engine Land summarized Perplexity’s position as saying its requests were user-initiated, real-time fetches rather than preemptive crawling.
Perplexity’s reproduced statement said: “When you misattribute millions of requests, publish completely inaccurate technical diagrams, and demonstrate a fundamental misunderstanding of how modern AI assistants work, you’ve forfeited any claim to expertise in this space.” That is advocacy from one side of a contested exchange, not a neutral technical conclusion. The direct Perplexity post was not available in the sources reviewed, so its arguments are presented here as reported by those publications.
Recommended Free Tools
Rank #2
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What did Cloudflare’s test actually test?
Discoverability
Cloudflare said the domains were newly purchased and not publicly indexed or otherwise discoverable. That was intended to test whether answers could be generated from content that ordinary search discovery would not expose.
Published crawler preferences
The sites included robots.txt directives disallowing the relevant crawlers. Cloudflare then added WAF rules intended to block the declared Perplexity crawler identities.
Traffic identification
Cloudflare said it compared user-agent strings, source IP ranges, autonomous systems and other network signals. It reported that traffic continued under a browser-like identity and changed network locations after blocks. Those are Cloudflare’s observations and interpretation of its own test.
Rank #3
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
What remains unresolved
The public accounts do not independently establish whether the browser-like requests were Perplexity-controlled fetching, BrowserBase traffic misattributed to Perplexity, user-triggered retrieval, preemptive crawling, or some combination. They also do not establish how the test content reached the answering system or whether the same behavior occurred across all sites using similar controls.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Does robots.txt actually stop AI bots?
No. robots.txt is a machine-readable protocol in which a site operator states which crawlers should avoid particular paths. It is a request to clients that choose to comply, not authentication and not a technical lock on a public URL.
A determined client can still request a publicly reachable page unless the site applies an enforcement mechanism. This is why Cloudflare’s description distinguishes the file from WAF rules. The two controls answer different questions:
Rank #4
- Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
- Identity: What user agent or network identity does the requester present?
- Preference: What does the site publish in
robots.txt? - Enforcement: Does a firewall, bot-management rule, authentication layer or access policy block or challenge the request?
- Permission: Is the fetching allowed under the site’s terms, contracts or applicable law?
Those layers should not be conflated. A request can violate a published crawler preference without being technically blocked, while a technically blocked request raises a separate question about how the requester attempted to proceed. The reports reviewed here do not resolve the broader legal issues.
Can a website block AI crawlers with a firewall?
A WAF or bot-management system can block, rate-limit or challenge requests using signals such as user-agent strings, IP reputation, behavioral patterns and network characteristics. Cloudflare said it removed Perplexity from its verified-bot list and added signatures for the traffic it observed to a managed rule intended to block AI crawling. It also said customers with existing bot-management block rules were protected and that challenge rules were available.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
These measures are not permanent guarantees. Cloudflare itself noted that crawler behavior and evasion methods can change, so the controls described were the company’s state at the time of its August 2025 post. Publishers using them need monitoring, updated signatures and a fallback plan for false positives affecting ordinary visitors or accessibility tools.
Best Value
- [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
Cloudflare’s separate controls explainer describes a managed robots.txt option that publishes directives against AI-training crawlers and is designed to update as the crawler landscape changes. That product description, like the reported adoption count, comes from Cloudflare and is not an independent assessment of effectiveness.
How the two accounts differ
| Question | Cloudflare’s account | Perplexity’s reported response |
|---|---|---|
| What happened after blocks? | Traffic continued through declared and browser-like identities after robots and WAF restrictions. | Some of the traffic was misattributed to BrowserBase’s automated browser service. |
| What was the purpose of fetching? | The pattern looked like crawling that obscured its identity after network blocks. | Requests were user-initiated, real-time retrieval rather than preemptive crawling. |
| How was traffic identified? | Cloudflare cited user-agent, IP, ASN, machine-learning and network signals. | Perplexity rejected Cloudflare’s technical interpretation. |
| What independent evidence exists? | Cloudflare published its test description and observations. | The sources reviewed contain the company’s rebuttal but no neutral replication. |
What this means for publishers
- Publish clear crawler directives, but do not treat
robots.txtas an access-control system. - Use WAF, bot-management or authentication controls when technical blocking is required.
- Log user-agent strings alongside source networks and behavior; any single signal can be spoofed or shared by legitimate services.
- Separate AI-training restrictions from real-time retrieval policies in your terms and enforcement rules.
- Review challenge and blocking rules for false positives, especially where browser automation supports legitimate users or accessibility workflows.
- Recheck rules periodically because crawler identities, hosting providers and evasion techniques can change.
Bottom line on the Cloudflare–Perplexity dispute
Cloudflare made a detailed, test-based allegation that Perplexity-related traffic reached blocked test sites and, in some cases, appeared to conceal its crawler identity. Perplexity rejected that account and attributed the reported traffic to misidentification and user-triggered browsing. Until independent evidence resolves those competing explanations, the accurate conclusion is that Cloudflare has alleged scraping or circumvention—not that the allegation has been proven across the web.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
