Cloudflare Pay Per Crawl Explained: Should a B2B Company Charge AI Crawlers?

In this article

1. What Pay Per Crawl is
2. How it works, step by step
3. Pricing rules worth knowing
4. What changed in 2026: Pay Per Use
5. Should a B2B company charge AI crawlers?
6. If you do want to charge: a safer setup
7. Frequently asked questions

What Pay Per Crawl is

Cloudflare launched Pay Per Crawl in July 2025 as a third option between letting AI crawlers in for free and blocking them. In AI Crawl Control, each AI crawler can be set to Allow, Block or Charge. A crawler set to Charge can still access your content, but only if it pays the price you set.

The idea answers a real problem for publishers. AI companies crawl huge amounts of content, and answers built from that content can reduce the visits that used to pay for it. Pay Per Crawl tries to put a price on the crawl itself.

How it works, step by step

1. A crawler requests a page set to Charge.
2. If it hasn't offered to pay, Cloudflare returns HTTP 402 Payment Required with a crawler-price header, for example crawler-price: USD 0.01.
3. The crawler retries with a crawler-exact-price header (agreeing to that exact price) or a crawler-max-price header (agreeing to pay any price up to a limit). The request must be signed with Web Bot Auth so Cloudflare can confirm who's paying.
4. Cloudflare serves the page with a crawler-charged header showing the amount billed.
5. Cloudflare bills the AI company and pays you. Cloudflare is the merchant of record. Payouts go monthly to a dedicated Cloudflare Stripe Connect account, subject to settlement periods and minimum thresholds.

What an unpaid crawler receives

HTTP/2 402
crawler-price: USD 0.01

Pricing rules worth knowing

• One price for all charged crawlers. You can't charge OpenAI one rate and Anthropic another. Each crawler is set to Block, Charge or Allow, and all charged crawlers pay the same price.
• Minimum $0.001 per crawl. You set the default price under AI Crawl Control → Payments.
• Dynamic pricing lets you set different prices for different content. Since June 2026 you can also switch Pay Per Crawl off for certain URL patterns, and set prices from your origin server or with Cloudflare Workers.
• Every successful fetch is charged, including re-crawls of the same page. Error responses aren't charged.
• Some files are always free: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json.
• It runs after Cloudflare's bot protections. If "Block AI bots" is on, charged crawlers never reach the payment step, so Cloudflare tells you to turn it off first.

What changed in 2026: Pay Per Use

On July 1, 2026, Cloudflare announced Pay Per Use, which TechCrunch described as the successor to Pay Per Crawl. Instead of charging for every fetch, publishers are paid "when their content creates value". For example, a publisher is paid when its content appears in a partner's AI search results. The two launch partners named were Ceramic.ai and You.com.

Cloudflare also said that over half of AI crawler traffic is spent re-fetching pages that haven't changed. That's one reason a per-fetch price is a blunt tool. On the same day it announced the Search, Agent and Training controls covered in our AI Crawl Control guide.

Should a B2B company charge AI crawlers?

Start with what your website is for. A publisher's articles are the product, and every AI answer that replaces a visit costs it ad or subscription revenue. A manufacturer's, distributor's or law firm's website is marketing. Its job is to make the company known, trusted and shortlisted. When ChatGPT or Perplexity reads your capabilities page and recommends you, that's the outcome you want.

Charging works against that outcome for three reasons:

1. Most crawlers won't pay. Cloudflare hasn't published a list of paying AI companies, and the named partners so far are small next to OpenAI, Google, Anthropic and Microsoft. For every crawler that doesn't pay, Charge works exactly like Block.
2. Blocked means invisible. If OAI-SearchBot or PerplexityBot can't read your pages, those assistants can't cite you. See what each type of AI bot does.
3. The money is small. At $0.01 per crawl, 10,000 paid crawls a month would earn $100. One B2B deal influenced by an AI recommendation is usually worth far more.

If you do want to charge: a safer setup

1. Keep AI search and assistant crawlers on Allow for all marketing pages.
2. Use dynamic pricing or URL-pattern rules so charging applies only to the folders that hold the content you sell.
3. Turn off "Block AI bots", otherwise charged crawlers are blocked before they can pay.
4. Connect Stripe through the Cloudflare dashboard. Existing Stripe accounts can't be reused, and an Administrator must set it up.
5. Watch two numbers for 60 days: AI crawler requests and 402 responses in AI Crawl Control → Metrics, and AI referral traffic in GA4. If referrals fall and revenue is negligible, switch back.

Frequently asked questions

Free tool

Make sure AI crawlers can read you first

Before deciding whether to charge AI crawlers, check that they can reach you at all. The free LLM Readiness Check tests 15 of them in 10 seconds.

Run the free check