Recommended Free Tools
Yes, but only in a limited and still-emerging sense. Cloudflare has built two products that let publishers attach a price to AI access: Pay Per Crawl, which charges for fetching a page, and Pay Per Use, which charges when an AI system reports that it has used content in a defined way. Both are described by Cloudflare as beta or private-beta programs, and neither has been shown to restore lost search or referral traffic. This article explains what each model does, how the two differ, and what a publisher should check before turning either on.
The headline’s story about one blogger’s traffic and earnings cannot be verified from the available material, so this piece treats the mechanisms and the market as the subject rather than any individual site’s numbers.
As an Amazon Associate I earn from qualifying purchases.
Crawler visits and human referral visits are different events
Most publishers who feel that AI has taken their traffic are describing two separate things. An AI crawler fetches your pages so a model or search product can index, train on, or retrieve them. A referral is a human who clicks through from an AI answer or search result to your site. A crawl does not count as a visit in your analytics, and a referral can disappear even when crawling continues. Cloudflare’s own framing distinguishes these: it argues that AI products fetch far more than they use, which is why its newer model prices use rather than access. Before you decide what to charge for, separate the two in your logs and analytics so you know which decline you are actually measuring.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Pay Per Crawl: charging for each fetch
Cloudflare announced Pay Per Crawl on July 1, 2025 as a private beta. In that announcement, Cloudflare said it would act as Merchant of Record and provide the underlying infrastructure, so the crawler operator pays Cloudflare rather than each publisher individually. The publisher controls three options for each crawler: allow, charge, or block.
| Publisher setting | What happens to a crawler | Notes from the launch announcement |
|---|---|---|
| Allow | The crawler receives the page without a charge. | Allowed crawlers are not billed. |
| Charge | The crawler must pay the publisher’s price per request. | At launch the price was a flat per-request rate across the whole domain. |
| Block | The crawler does not receive the content. | Publishers could make exceptions for specific crawlers, including bypassing charges for selected ones. |
How the payment handshake works
The announcement described two ways a paying crawler signals intent. It can send payment intent in its request headers. Or it can request a page, receive an HTTP 402 Payment Required response carrying a crawler-price header, and retry the request at the agreed price. The announcement also described identity checks based on Web Bot Auth-related signatures, which are meant to stop a crawler from simply claiming to be a different bot. Because the flow depends on signed identity, a publisher’s ability to charge a crawler is only as good as the identity check behind it.
Full details are in Cloudflare’s launch post at https://blog.cloudflare.com/introducing-pay-per-crawl/. Treat its pricing and controls as the July 2025 design, not a fixed set of current product terms.
Pay Per Use: charging when the AI uses your work
Cloudflare’s later Pay Per Use model moves the payment trigger from the fetch to the downstream use. In its description, the AI buyer defines the use it wants to pay for, sets an offer price, and reports each use it makes. The publisher decides whether to participate in a given offer. Cloudflare gives two examples: an excerpt of your article shown in a search result, and a review of your product that influences a shopping agent’s recommendation.
Cloudflare says it handles enrollment, usage records, billing, and monthly settlement to publishers. The article also states that usage is self-reported by the buyer, that buyers must report complete usage under the program’s terms, and that Cloudflare checks that each reported use maps to an enrolled publisher. The product is described as beta, so it is reasonable to expect limited participation and not to assume a steady income stream. The original article is at https://blog.cloudflare.com/pay-per-use/.
The two models side by side
| Question | Pay Per Crawl | Pay Per Use |
|---|---|---|
| What triggers payment? | A crawler fetching a page | A buyer reporting a defined downstream use of content |
| Who sets the price? | The publisher, as a flat per-request price across the domain at launch | The buyer sets an offer price; the publisher accepts or declines |
| Publisher control | Allow, charge, or block, with per-crawler exceptions | Participation in each offer; the buyer defines the use |
| Identity | Web Bot Auth-related signatures described at launch | Buyer identifies itself and reports use; Cloudflare checks reported uses against enrolled publishers |
| Reporting | Not described in the launch announcement | Self-reported by the buyer under program terms |
| Settlement | Cloudflare acts as Merchant of Record; settlement terms not stated in the launch announcement | Cloudflare handles billing and monthly settlement |
| Stated status | Private beta (July 2025 announcement) | Beta (later Cloudflare description; date not stated in the source) |
The two models answer different questions. Pay Per Crawl asks whether a bot should pay to access a page. Pay Per Use asks whether a company should pay for what it does with the content. A publisher worried about model training may find the crawl model more relevant, while one worried about AI answers that replace clicks may find the use model closer to the problem. Neither model replaces ads, subscriptions, licensing deals, or direct audience building.
Who is paying, and how big the market is
Cloudflare’s July 1, 2026 press release says more than 50 major content-licensing agreements were signed over the preceding year. That is Cloudflare’s own count and characterization, and it does not establish how many publishers are enrolled in its products. The release names two AI-related partners as examples of the emerging market. Ceramic.ai is described as paying when an opted-in publisher’s content appears in AI search results, with queries, citations, and ranking returned. You.com is described as letting an agent pay on demand for specific premium content. The release also names beehiiv as a platform partner that provides dashboards for tracking, blocking, or opting into AI models. These are examples of what Cloudflare is building, not evidence of a general signup path or of commercial terms available to any given site. The release is at https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/.
Rank #4
What the crawler data says, and how far to trust it
Cloudflare’s 2026 report on the first year after its Content Independence Day initiative says that, in its own classification of crawler requests by purpose, 52% were for AI training in June 2026. That is up from 22% in Spring 2025. Mixed-use crawlers, which do not separate their purposes, accounted for over 36% of activity. Cloudflare also says pure search crawling is a small and declining share of crawler activity, while remaining important for visibility. These are Cloudflare’s measurements and labels from its own network, and they should not be read as the proportion of all internet requests. The report is at https://blog.cloudflare.com/agentic-internet-bot-report/.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Matthew Prince, Cloudflare’s co-founder and CEO, said in the July 2026 announcement: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” That is a statement of the company’s position, not an independently validated traffic statistic.
Best Value
The default change scheduled for September 15, 2026
Cloudflare’s July 1, 2026 release announced a default change for existing free customers who had not altered their settings. From September 15, 2026, search crawling would be allowed on new sites and new domains, while training and agent use on pages carrying ads would be blocked. The release also said mixed crawlers that do not separate their purposes would be blocked on ad-supported pages. Customers could change these settings in their dashboard.
That date has now passed. The announcement does not establish how the change was applied to any particular account, so check your own Cloudflare dashboard to see which crawlers are allowed, charged, or blocked on your domain before you assume any default.
Decisions to make before turning anything on
- Separate the traffic types. Compare crawler activity with human referrals over the same period so a fall in visits is not blamed on crawling alone.
- Decide by purpose, not by name. Search crawlers, training crawlers, and mixed crawlers have different value to you. Blocking a search crawler may cost visibility that Cloudflare itself describes as important.
- Check the terms of any buyer. Pay Per Use depends on buyers self-reporting usage. Ask what evidence you will see for each reported use and how disputes are handled.
- Confirm settlement and availability. Both products are described as beta or private beta, and the sources do not state which accounts are eligible or how often payments are settled beyond the monthly settlement Cloudflare describes for Pay Per Use.
- Plan for a small return. Nothing in the available material establishes revenue levels, pricing that works for a typical blog, or any effect on rankings or referrals. Treat payments as an experiment that may not cover the traffic you lose.
Known limits
- Crawler identity depends on signatures. A bot that spoofs identity could evade a paywall, which is why the identity check is central to both models.
- Pay Per Use depends on buyers reporting their own use. Cloudflare checks that reported uses map to enrolled publishers, but the material does not describe independent audits of the reported counts.
- Blocking an AI product can also remove your content from answers that send readers to you, so the trade-off is not one-sided.
- Vendor figures on crawler shares come from Cloudflare’s own network and classification, not from a neutral industry count.
The practical answer for a blog is to measure first, decide crawler by crawler, and treat either payment model as an optional, beta-stage revenue line rather than a fix for lost traffic.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
