AI Crawl Control is Cloudflare’s main dashboard for deciding whether supported AI crawlers may access your content, are blocked, or are offered a paid-access flow. Configure crawler actions in AI Crawl Control and Pay Per Crawl in its Payments tab. A 402 response can mean either a block configured to say “Payment Required” or the first step of Pay Per Crawl; only the latter uses the signed payment-header exchange to authorize a paid retrieval.
Which Cloudflare setting should you use?
| Goal | Setting or mechanism | What it does |
|---|---|---|
| Permit or refuse a crawler | AI Crawl Control action: Allow or Block | Sets the crawler’s access policy. A block can return 403 Forbidden or 402 Payment Required, with a custom plain-text response body. |
| Offer supported AI crawlers paid access | AI Crawl Control → Payments → Pay Per Crawl | Quotes a price with HTTP 402, then permits a successful paid retrieval when the crawler retries with an acceptable signed payment header. |
| Keep particular routes free or charge different amounts | Configuration Rules or dynamic crawler-price |
Exempts matching paths from Pay Per Crawl or lets an origin response set a per-request price instead of the zone default. |
| Build payment gates for selected routes | Cloudflare Workers x402 Payment-Gated Proxy template | Provides a separate Worker-based route-gating approach using the x402 protocol; it is not a Pay Per Crawl dashboard toggle. |
These controls do not operate in isolation: WAF custom rules and Cloudflare bot settings can act before Pay Per Crawl and prevent a crawler from reaching the payment flow.
As an Amazon Associate I earn from qualifying purchases.
Set a crawler’s access policy
In AI Crawl Control, review crawler activity and choose an action for each crawler:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Allow: Permit access subject to other zone controls and robots.txt enforcement.
- Block: Refuse access. Choose 403 Forbidden or 402 Payment Required and, if useful, provide a custom plain-text message.
- Charge: Direct an eligible crawler to the Pay Per Crawl flow, if the feature is enabled for the account.
Cloudflare’s block-response code selector supports 403 and 402. Manually changing the associated WAF rule to a different response code can prevent AI Crawl Control from enforcing the selected code. An Allow action is not a general bypass of WAF rules, bot policies, or robots.txt.
#1 Best Overall
Enable Pay Per Crawl and set the default price
- Open AI Crawl Control → Payments → Pay Per Crawl.
- Select Enable, set the default price, and save.
Cloudflare documents Pay Per Crawl as a private beta, so availability depends on account eligibility. Its documentation gives a minimum of USD $0.001 per crawl and says the configured amount is charged for each successful HTTP 200 content retrieval. Treat the minimum and beta status as subject to change; see Cloudflare’s Pay Per Crawl pricing documentation.
What happens in the Pay Per Crawl 402 exchange?
- Initial request: A crawler requests a page protected by Pay Per Crawl. Cloudflare responds with HTTP/2 402 Payment Required and a
crawler-priceheader quoting the amount. - Payment retry: To proceed, the crawler retries with either
crawler-exact-price, matching the quoted amount, orcrawler-max-price, covering it. The payment header must be included among the Web Bot Authsignature-inputcomponents. - Successful retrieval: Cloudflare returns HTTP 200 with
crawler-charged. The crawler operator should track that confirmation. - Failed paid request: A failed request can return 402 with a
crawler-errorheader.
A 402 by itself does not collect payment. The crawler must support the onboarding and Web Bot Auth requirements and send the prescribed signed header. Cloudflare’s crawler-flow documentation describes the request and response headers.
Distinguish a 402 block from a payment offer
Both configurations can produce HTTP 402, but their purpose differs. A Block action with a 402 response tells the crawler it is refused under the site’s policy; a custom message can explain the condition. It does not by itself start a charge or authorize access. Pay Per Crawl uses 402 as a price quote in a handshake: the crawler can retry with the signed payment header, and a successful paid retrieval returns HTTP 200 with crawler-charged.
Check WAF and bot-rule precedence if charging does not work
Cloudflare documents this processing order:
- AI Crawl Control crawler-block WAF rules
- Cloudflare bot solutions
- AI Crawl Control Pay Per Crawl
Consequently, a WAF block or bot action can stop the request before Pay Per Crawl is reached, so no charge flow occurs. Cloudflare specifically says to turn off the Block AI Bots bot configuration when the intention is to use Pay Per Crawl. WAF custom rules can also affect crawlers marked Allow. Review those controls separately rather than assuming an AI Crawl Control decision overrides them. See Cloudflare’s WAF integration guidance.
Exempt routes or set different prices
Make selected paths free
Create a Configuration Rule under Rules, match the relevant path with a URI Full condition, and enable Disable Pay Per Crawl. Cloudflare’s guidance recommends preserving free access to navigation and discovery pages, while considering exemptions for functional endpoints such as login, search, and APIs. A homepage exact match or public-directory wildcard can be used as the URI pattern, as appropriate to the site’s routes.
Cloudflare lists /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, and /crawlers.json as always free.
Override the zone default dynamically
When dynamic pricing is enabled, an origin response can set a crawler-price header, for example crawler-price: USD 3.14. That value replaces the zone’s default for the request, and Cloudflare forwards the price header to the client when it is being asked to pay. This supports different prices for different content without changing the default for every crawler.
When a Worker x402 template is the better fit
Cloudflare also documents an x402 Payment-Gated Proxy Worker template. It is an alternative when you need payment logic on selected routes—for example, gating a route for bots while allowing human visitors free access. It uses the x402 protocol and HTTP 402 and requires a Worker deployment and protocol setup; it is distinct from enabling Pay Per Crawl in AI Crawl Control. See Cloudflare’s x402 Payment-Gated Proxy template documentation.
What to check when a crawler receives 402
- Confirm whether the crawler’s action is Block with a 402 response or Charge through Pay Per Crawl.
- For a payment offer, inspect the
crawler-pricequote and whether the crawler retries with a supported, correctly signed payment header. - For a successful paid response, look for HTTP 200 and
crawler-charged; a failed payment request can includecrawler-error. - Check WAF custom rules and bot settings for an earlier block, including the Block AI Bots configuration.
- Check whether a Configuration Rule disables Pay Per Crawl for the requested URI.
- Verify that the crawler is eligible and onboarded for the documented Web Bot Auth payment flow; not every bot supports it.
Cloudflare says successful repeat crawls incur the configured cost again. The five listed discovery and security files remain free under its documentation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
