What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Cloudflare is not making every AI company pay to crawl the web. Its optional Pay Per Crawl feature lets participating website owners allow, charge, or block identified AI crawlers. A crawler that encounters a price can pay through an HTTP-based process to retrieve the content. The payment feature was still in closed beta in Cloudflare’s documentation last updated July 28, 2026; broader AI Crawl Control tools, including crawler monitoring and controls, are available on all Cloudflare plans.
Why Cloudflare is proposing payment for AI crawls
Traditional search crawling has often been part of an informal exchange: a search engine indexes a site and may send people back to it. Generative AI systems can use crawled material without necessarily sending a comparable number of visitors to its source. Cloudflare argues that this weakens the old referral bargain and gives publishers a reason to seek compensation. That is Cloudflare’s framing of the market, not a description of every AI-content arrangement: some material is covered by direct agreements, supplied datasets, or other terms.
To illustrate the imbalance, Cloudflare reported that in June 2025 it observed about 1,700 OpenAI crawler requests and 73,000 Anthropic crawler requests for each referral, respectively. These are Cloudflare’s measurements for the period it described, not an independent audit or a census of all websites. Cloudflare has also reported sharp year-over-year growth in crawler activity on its network, including a 305% increase for GPTBot and 96% for Googlebot between May 2024 and May 2025. Those figures describe traffic Cloudflare observed, not the entire web. Cloudflare’s account of its crawler and referral measurements explains its methodology and rationale.
Free tools Windows power users keep installed
One-click scans. No signup required.
Cloudflare introduced Pay Per Crawl on July 1, 2025. Subsequent additions documented in 2026 include crawler discovery, signed payment headers, URI-based exemptions, dynamic pricing, and expanded analytics. The feature remains a beta service rather than a settled, universally adopted payment system.
#1 Best Overall
How a paid crawl works
The transaction is built around ordinary HTTP requests and responses, with crawler authentication and payment-intent headers added to the exchange. A simplified example is below; the price is illustrative of Cloudflare’s documented minimum, not a recommended rate.
- The crawler requests a page. It first sends a request without agreeing to pay. The site’s Cloudflare configuration determines that this crawler should be charged.
- The site returns a price. Instead of serving the protected content, it responds with
HTTP 402 Payment Requiredand acrawler-priceheader, for examplecrawler-price: USD 0.01. - The crawler decides whether to proceed. If it accepts, it retries with a payment-intent header:
crawler-exact-price: USD 0.01to accept that price, orcrawler-max-price: USD 0.05to set a ceiling. - The payment intent is authenticated. The payment header must be covered by the request’s Web Bot Auth signature. This helps prevent someone from tampering with the stated payment terms.
- The site serves the content and reports the charge. If the price and authentication are valid, the request can receive
HTTP 200and acrawler-chargedheader identifying the amount. The crawler should record that value to track its spending. - Cloudflare handles the transaction layer. Cloudflare records and aggregates charges and acts as merchant of record; publishers connect a dedicated Stripe Connect account for payouts.
The documented minimum publisher-set price is $0.01 per successful retrieval. Cloudflare has not published a universal recommended price or a public platform-fee schedule for this feature. The protocol and configuration are described in Cloudflare’s Pay Per Crawl overview and its crawler implementation guide.
What website owners can choose
A publisher configures the feature in Cloudflare’s AI Crawl Control dashboard. The basic workflow is to enable Pay Per Crawl, set a default price, choose an action for each crawler, and connect the payout account. The available actions are:
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall- Allow: let the crawler access content without a Pay Per Crawl charge.
- Charge: return the price-and-payment flow for content requests from that crawler.
- Block: deny access rather than offer a paid request.
The standard dashboard price applies to all crawlers assigned the Charge action; the basic configuration is not a separate negotiated rate for each AI company. Cloudflare’s later advanced-configuration documentation describes URI-based exemptions and dynamic pricing based on request properties, including pricing determined at the origin or by a Cloudflare Worker. These controls can make a site’s policy more granular, but they do not turn Pay Per Crawl into a marketplace for individually negotiated content licenses. See the price configuration guide, crawler action guide, and advanced-configuration changelog.
Some infrastructure paths are free to crawl under the documented rules: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, and /crawlers.json. URI exemptions can also be useful for licensing information or other pages a publisher wants to leave freely accessible. Cloudflare’s Pay Per Crawl FAQ says successful content retrievals are billed, error responses are not, and retrieving the same page again can trigger another charge.
Which bots reach the payment step?
Pay Per Crawl does not override earlier security decisions. Cloudflare says WAF and bot-management rules run before the payment feature; if an earlier rule blocks a request, it never reaches the charge stage. A publisher seeking payment therefore needs to review rule ordering and ensure relevant requests are allowed through those earlier controls. Cloudflare’s bot configuration guidance describes how these controls interact.
What AI crawler operators need to support
A crawler cannot be charged through this flow merely because a site owner labels it an AI bot. The operator needs an onboarded, identifiable crawler identity, Web Bot Auth support, and code that understands the 402 response and payment headers. It must decide whether to accept a quoted price, sign the payment intent, retry, and keep an accurate record of the charge. Cloudflare’s documentation also describes a discovery API through which verified crawlers can find participating domains.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →For an operator, the practical controls are as important as the payment handshake:
- Set price ceilings so an unexpected quote cannot exceed the crawler’s budget.
- Cache results and deduplicate URLs; repeat retrievals may be charged even when the underlying page has not changed.
- Track the
crawler-chargedamount rather than assuming the quoted price is the final accounting record. - Define handling for redirects, partial responses, retries, rate limits, and origin failures. Cloudflare documents that error responses are not billed, but that does not establish one universal outcome for every failure pattern.
The protocol therefore rewards participation by crawler operators that identify themselves and follow the payment process. It is not a mechanism that can reliably identify or collect money from every automation tool on the internet.
What the payment does—and does not—buy
A per-request charge is a transaction for access under the site’s configured rules. By itself, it does not define the scope of a copyright license or settle whether a particular use is legally permitted. Nor does it specify what an operator may do with retrieved material after access.
Publishers that need terms for model training, storage, redistribution, attribution, deletion, duration, geographic scope, derivative works, indemnification, or audit rights may need a separate agreement. Pay Per Crawl can provide an access-control and payment layer; it is not a substitute for every direct licensing arrangement, archive deal, or content-specific contract.
How publishers receive money—and what revenue is uncertain
Cloudflare aggregates charges and pays participating publishers through Stripe. The documented setup requires a dedicated Stripe Connect account for Pay Per Crawl; an existing Stripe account is not compatible with this feature. Cloudflare describes monthly payouts for publishers in good standing, subject to settlement periods and minimum payout thresholds. Its documentation does not state the specific Stripe economics for this beta integration. The setup details are in Cloudflare’s payout guide.
Revenue depends on actual paid volume, price, crawler participation, repeat requests, and the kinds of content a publisher makes chargeable. At $0.01 per successful retrieval, one million such retrievals would total $10,000 before any applicable commercial terms, taxes, settlement effects, or other deductions. That is arithmetic, not a forecast: Cloudflare does not publish a general revenue estimate, and the beta does not establish how many major AI operators will pay. A low fee might still produce high-volume traffic with little compensation; a high fee might lead crawlers to skip the material.
Pricing also has to account for URL structure. Query strings, duplicate URLs, pagination, feeds, images, APIs, and crawler retry behavior can create many billable requests for what a publisher considers one piece of content. URI rules, canonicalization, cache policy, and usage analytics should be considered before applying a charge broadly.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Search visibility and crawler identity are real constraints
“AI crawler” is not one interchangeable category. Search indexing, AI-assisted search, model training, retrieval, and answer generation can involve different crawlers and business outcomes. Cloudflare warns that charging or blocking search-engine crawlers may hurt SEO if they cannot index the site. Publishers should identify a crawler’s operator and function before choosing Allow, Charge, or Block, especially when a bot may contribute to search discovery. Cloudflare’s crawler-selection guidance discusses the SEO concern.
Identification is also a limitation. The payment workflow depends on recognized, authenticated crawler traffic. Undeclared or spoofed bots, residential proxies, browser-based automation, and other evasive scraping may not use the protocol honestly. Pay Per Crawl is best understood as a control and transaction path for participating traffic, not a guarantee that all scraping can be detected, stopped, or monetized.
How to decide whether it fits a site
For a publisher considering beta access, the decision is less about whether charging sounds appealing than whether the site can define and measure a workable policy.
- Value: Are the pages proprietary, expert-produced, current, or difficult to replace, or is their value mainly in attracting human readers?
- Discovery: Would reduced indexing or AI-search access cost more than likely crawler revenue?
- Segmentation: Can the site separate premium research or specialized data from ordinary editorial pages, and set URI or dynamic rules accordingly?
- Identity: Are the crawlers it wants to charge identifiable operators likely to implement Web Bot Auth and payment handling?
- Operations: Can staff monitor analytics, review WAF and bot rules, handle false positives, and respond to crawler or customer questions?
- Legal fit: Does a request-level payment meet the business need, or is a negotiated license still necessary?
- Measurement: Will the publisher track revenue, referral traffic, crawler mix, bandwidth, and exposure separately rather than treating any one metric as success?
A sensible rollout, if a site has beta access, is to begin with a narrow content category and a small set of known crawlers. Keep search crawlers free where indexing matters, exempt necessary discovery and licensing paths, review upstream security rules, and watch for repeat or duplicate requests before expanding the policy. This is a risk-management approach, not a promise that the feature will produce meaningful income.
What Pay Per Crawl means for the web
The idea occupies a middle ground between allowing all crawler access for free and blocking it outright. Cloudflare supplies visibility, controls, the HTTP payment flow, billing aggregation, and payout infrastructure, while publishers set the policy and crawler operators decide whether to participate. That can reduce the engineering burden of handling each transaction independently, but it also makes Cloudflare a central intermediary in the payment relationship.
The model has a network-effects problem: publishers need AI operators to support the protocol, while operators need enough valuable participating content to justify implementation and payment. Direct licenses may suit large buyers and publishers better for broad training rights; request-level pricing may be more practical for metered access to specific pages. Neither approach automatically solves the other’s needs.
The feature is commercially interesting because it makes paid access technically concrete, but its beta status, reliance on crawler cooperation, repeated-request billing, and unresolved licensing questions leave its broader economics unproven. Cloudflare describes its wider AI Crawl Control availability and product controls at the AI Crawl Control documentation; the Pay Per Crawl overview carries the current beta-status details.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
