Cloudflare is building a way for website operators to charge participating AI crawlers for access to their pages. That is narrower than being paid whenever an AI model trains on “your data.” Pay Per Crawl charges for a successful retrieval at a Cloudflare-protected domain; it does not identify past training use, create automatic royalties, or pay every individual whose work appears online.
As of Cloudflare’s August 18, 2026 documentation, AI Crawl Control’s monitoring and access policies are available on all Cloudflare plans, while Pay Per Crawl remains in closed beta. Eligibility, crawler participation and actual demand determine whether the system produces any revenue.
As an Amazon Associate I earn from qualifying purchases.
What Cloudflare is actually offering
Cloudflare’s AI Crawl Control lets a site operator identify AI services visiting a domain and choose whether each crawler is allowed, blocked or charged. The product is intended to address a changed web bargain: search engines traditionally sent visitors back to publishers, while AI systems can retrieve and summarize material without generating comparable referral traffic.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCloudflare reported that, in its June 2025 measurements, OpenAI crawlers made about 1,700 requests per referral and Anthropic crawlers about 73,000 requests per referral. Those are Cloudflare’s observations, not universal industry ratios. (Cloudflare’s crawler analysis)
#1 Best Overall
- WIFI ENABLED TO CONTROL FROM ANYWHERE – Transform your home into a smart home with the Feit Electric Smart Wi-Fi Plug. Remotely turn on or off lights, fans, coffee makers, or other home appliances from your smartphone or tablet. Works seamlessly with Alexa and Google Home, giving you effortless voice control without needing a separate hub. Manage your devices anytime, whether you’re at home, at work, or traveling.
- SIMPLE SETUP, NO HUB REQUIRED – Enjoy the convenience of smart home automation without extra equipment. The plug connects directly to your 2.4 GHz Wi-Fi network, making installation fast and easy. Plug it in, download the Feit Electric app, follow the simple steps, and your devices are instantly connected. Perfect for beginners or anyone looking to expand their smart home ecosystem with minimal hassle.
- SET YOUR ROUTINE & SAVE ENERGY – Save energy, stay organized, and automate daily routines with customizable schedules and timers. Set your lamps, heaters, or appliances to turn on and off automatically at specific times, ensuring your home is always comfortable and efficient. Ideal for morning routines, evening wind-downs, or holiday lighting, giving you peace of mind and energy savings without constant manual operation.
- ENHANCED SAFETY & CONVENIENCE – Protect your home and appliances with the Feit Electric Smart Plug’s durable design and safety features. Its compact size fits easily into standard indoor outlets without blocking other sockets. With real-time app control and notifications, you can monitor appliance activity and prevent energy waste. Ideal for families, pet owners, or anyone seeking a smarter, safer, and more convenient home setup.
- RELIABLE 2.4GHz WI-FI PERFORMANCE – Designed to work exclusively on 2.4 GHz networks, this smart plug provides stable connectivity for smooth operation of all your devices. Avoid interruptions caused by incompatible networks, ensuring your appliances respond instantly when controlled via the app or voice commands. Perfect for indoor home use, it supports up to 15 amps, handling heavy-duty appliances safely and reliably.
The company’s proposed choices are straightforward:
- Allow the crawler for free.
- Block it.
- Allow it only when it pays.
Pay Per Crawl is the payment option. Cloudflare’s broader AI Crawl Control documentation covers visibility, policy controls and robots.txt monitoring.
Current availability and timeline
| Date | Development |
|---|---|
| July 1, 2025 | Cloudflare announced Pay Per Crawl as a private-beta proposal. (Announcement) |
| July 28, 2026 | Cloudflare documentation still described Pay Per Crawl as closed beta. (Current product status) |
| July 1, 2026 | Cloudflare described Pay Per Use experiments intended to tie compensation to value such as appearance in an AI search result. (Pay Per Use announcement) |
| September 15, 2026 | Cloudflare announced a future default for new customers and newly created sites: on pages displaying ads, search crawlers would be allowed while training and agent-use crawlers would be blocked. This was an announced change, not the default described in the August 18 documentation. (Cloudflare announcement) |
Cloudflare’s AI Index is also presented as private beta or early access. It is not an established marketplace with guaranteed buyers or published earnings rates.
How Pay Per Crawl works
The site-owner setup
- Put the domain behind Cloudflare.
- Open AI Crawl Control and review crawler activity.
- Set an allow, block or charge policy for identified crawlers.
- If accepted into the closed beta, enable Pay Per Crawl and set a default price.
- Optionally add URI-pattern exceptions or dynamic pricing.
- Connect the dedicated Stripe account from
Manage Account > Settings > Pay Per Crawl > Connect. - Monitor successful deliveries and payout records.
The documented minimum price is $0.001 per crawl. It is a floor, not an estimate of revenue. (Price configuration)
Rank #2
- equipped with atom n2600 d2700 processor, compatible with many freebsd based router systems, linux distros, or win.os supported, easy configuration and management
- Please note, this is a barebone only. A system memory, a storage drive and an operating system are needed to complete this system
- 13-19 inches 1u, 50w power, with power cord, make sure to use a big brand memory and ssd/hdd with quality assurance
- Designed with console, 2 x usb, 4 x lan, vga, power switch, size at 290 x 180 x 44mm
- There are 2 inside reserved fans on chassis, which could be removed freely or be turned on in a high temperature environment to ensure the best function of the product
The crawler request sequence
- The crawler requests a protected page.
- Cloudflare returns
HTTP 402 Payment Requiredand a price, for examplecrawler-price: USD 0.01. - The crawler retries with
crawler-exact-priceorcrawler-max-price. - Those payment headers must be included in the cryptographically signed Web Bot Auth request.
- If authentication and payment succeed, the page is delivered with
HTTP 200and acrawler-chargedheader. - Failed requests are not billed.
The full protocol is documented in Cloudflare’s crawler instructions. A bot cannot gain access by inventing a payment header; it must support Cloudflare’s onboarding, authentication and signed-payment flow.
Billing and payout
Cloudflare acts as merchant of record and manages the billing lifecycle. A publisher must create the dedicated Stripe Connect account through Cloudflare; an existing unrelated Stripe account is not compatible. Cloudflare documents aggregated payouts, described as monthly for publishers in good standing, with settlement periods and minimum thresholds that can vary and are not specified on the cited setup page. (Payout documentation)
What is—and is not—being paid for
| Event | Pay Per Crawl treatment |
|---|---|
| Successful retrieval of a protected page | Charge can be recorded. |
| Proof that a model trained on the page | Not established by the transaction. |
| Citation in an AI answer | Not what the current per-crawl charge measures. |
| Tokens extracted, content uniqueness or copyright status | Not the billing unit. |
| Future model revenue | No automatic royalty calculation. |
A training crawler might pay and retrieve a page, but the transaction occurs at the access layer. Cloudflare does not verify whether the crawler later trains a model, uses the page for retrieval, stores it in a dataset or discards it. Nor does payment alone determine whether a copyright license exists.
Cloudflare’s later Pay Per Use experiments acknowledge that one fetch may not represent the value created when content appears in an AI search result or supports an agent. Those experiments remain experimental rather than a generally available compensation system.
Who should consider it?
Potentially suitable operators
- Publishers, documentation providers and businesses with original, frequently updated or commercially valuable information.
- Operators already using Cloudflare that want crawler-level logs and policy controls.
- Organizations willing to trade some AI access for payment and test demand during a beta.
- Owners with clear authority to license every material element returned from the charged pages.
Likely poor fits
- Creators who cannot control HTTP responses because their work is hosted on another platform.
- Sites that need unrestricted free crawling for discovery and cannot tolerate mistaken blocks.
- Operators expecting compensation for historical model training or royalties based on answer usage.
- Domains containing substantial user-generated, syndicated, stock or otherwise restricted material.
- Businesses unable to complete Cloudflare and Stripe verification.
Platforms hosting user content need special care: owning the domain does not necessarily give the platform authority to charge for every contributor’s words, images or videos.
Rank #3
- Shelly Plus 1 PM is a Wi-Fi smart relay switch with 1 channel, up to 16A with power metering that can be used also as a WiFi repeater and Bluetooth gateway. Shelly Plus 1PM can be used to monitor the consumption and take control of home appliances, electric circuits, and office equipment individually.
- Automate electrical appliance and control - With Shelly Plus 1PM you can automate any electrical appliance in your home and control it remotely. Shelly Plus 1PM can control appliances with a large load which makes it perfect for kitchen appliances and domestic systems monitoring and control. You can get precise measurements of the power consumption of each appliance and switch in on/off remotely, no matter where you are.
- Set and be prepared for everything - Reveal the full potential of Shelly Plus 1PM by combining it with other devices from your home network! Set Shelly Plus 1PM to activate custom scenes based on hour, light, or various occurrences. For example, you can set Shelly Door/Window sensor to report a porch door opening and activate Shelly Plus 1PM to turn on the hot tub heaters only in the hours after 8 pm.
- Shelly Customer Service - Shelly is one of the fastest-growing Smart Home brands in the world with devices, providing solutions for the automation of private homes, buildings and businesses. We provide our customers with professional support and a 3 years device warranty.
- Shelly Smart Control App will help you control your Shelly devices remotely and will send notifications for all automated events in your home. You can easily configure devices and manage their settings individually, or you can create personalized scenes by combining Shelly devices to trigger certain actions in your home automation.
How much could a site earn?
At the documented floor, one million successful paid crawls would equal $1,000 before applicable commercial, settlement or tax considerations. That is arithmetic, not a forecast. Actual results depend on how many crawlers visit, whether they support the protocol, whether they accept the site’s price and whether repeated requests are charged.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteCloudflare’s FAQ indicates that repeated access can create repeated charges. A recrawl, a stale-page fetch and a first retrieval may therefore be separate billable events unless the publisher uses exceptions or dynamic pricing. The same FAQ says the current standard setup applies one price to all crawlers selected for charging. (FAQ)
Cloudflare’s June 2026 update added URI-pattern exceptions, dynamic pricing, an origin or Worker-provided crawler-price response header and a cf-pay-per-crawl request header. Beta documentation should be checked before relying on those controls. (Configuration update)
Important limits and failure modes
AI companies can refuse
Cloudflare cannot force an AI company to crawl, pay, accept a price or use the protocol. An unsupported crawler may stop, retry unpaid or require a separate WAF or Bot Management rule.
Blocking rules override charging
If WAF or Bot Management blocks a crawler, that rule takes precedence over Pay Per Crawl. The crawler receives no page and cannot create a paid retrieval. (Product behavior)
Recommended Free Tools
Rank #4
- Portable 100M/1G Network TAP Appliance for remote capture of data traffic
- Integrated with a Raspberry Pi 4 module (8GB RAM and 64GB Micro SD Card)
- Can be used as a standalone 100M/1G network TAP with the external monitor port
- Dual DC power inputs for enhancing overall system availability
Classification can be imperfect
Search, training and agent functions can overlap, and crawler identities can change. Test policies against search visibility, referrals and site functionality before broad enforcement. Cloudflare’s AI Crawl Control launch explanation describes the monitoring and classification approach.
Free policy files remain accessible
The FAQ lists /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json as always-free paths. They provide policy and discovery information rather than paid content access.
Payment is not a legal settlement
Pay Per Crawl does not resolve fair-use questions, copyright ownership, statutory licensing, dataset scope or the rights of third parties. Cloudflare’s sample AI-related terms can help explain a site’s preferences, but they are not a complete legal strategy. Obtain advice for the relevant jurisdiction. (Sample terms)
How it compares with other strategies
| Approach | Strength | Limitation |
|---|---|---|
| Cloudflare Pay Per Crawl | Controls future requests and can automate charges for compatible crawlers. | Closed beta; depends on crawler adoption; fetches are not downstream royalties. |
| Blocking and robots.txt | Expresses or enforces a no-crawl preference. | Creates no payment stream and cannot erase existing copies. |
| Paid API | Offers authentication, schemas, quotas and clearer usage records. | Requires engineering and ongoing maintenance. |
| Direct licensing | Can define datasets, duration, training, attribution, exclusivity and compensation. | Negotiation is slower and usually favors parties with commercial leverage. |
| Licensing collective or marketplace | May aggregate rights and negotiate standardized deals. | Terms, coverage and buyer demand vary by service. |
| Structured feed or syndication | Can provide controlled, purpose-built delivery. | Requires a separate commercial relationship and distribution workflow. |
A practical decision checklist
- Do you control the domain, edge responses and crawler policy?
- Do you own or have permission to license every material element on charged pages?
- Would losing unpaid AI access harm search discovery, referrals or customer service?
- Can the AI crawlers that matter authenticate and pay through Cloudflare’s protocol?
- Can your organization complete Stripe business and banking verification?
- Would an API or negotiated license better describe the value and permitted uses?
- Have you tested mixed-use crawler rules and confirmed that WAF policies do not override charging?
Bottom line
Cloudflare is creating a transaction and enforcement layer for AI-era web access, not a retroactive royalty fund. Pay Per Crawl may let eligible website owners charge compatible crawlers for successful page retrievals, but it remains closed beta, has no guaranteed buyers or earnings benchmark, and does not prove training use. For a publisher with valuable content and a willingness to experiment, AI Crawl Control is useful even without monetization; for defined datasets, historical use or model-level compensation, a paid API or direct license remains the more precise instrument.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
