Cloudflare WAF and Bot Rules for AI Crawlers: A Step-by-Step Setup

In this article

1. How Cloudflare's layers stack up
2. Step 1: Find out what's being blocked today
3. Step 2: Set crawler actions in AI Crawl Control
4. Step 3: Deal with Bot Fight Mode (Free plan)
5. Step 4: Let verified search crawlers skip bot protection and rate limits
6. Step 5: Block bots that pretend to be AI crawlers
7. Step 6: Block training crawlers only where you need to
8. Step 7: Rate-limit the unverified scrapers
9. Step 8: Verify and keep an eye on it
10. The full rule set at a glance
11. Frequently asked questions

How Cloudflare's layers stack up

Before writing rules, it helps to know the order in which Cloudflare evaluates a request from an AI crawler. Cloudflare's documentation describes this sequence:

1. WAF custom rules, including the rule called AI Crawl Control that the dashboard creates when you block a crawler (under Security → Security rules).
2. Bot solutions: Bot Fight Mode, Super Bot Fight Mode or Bot Management, depending on your plan.
3. Pay Per Crawl, if you use it.

Separately, the AI traffic settings under Security → Settings (Search, Agent and Training, added in July 2026) control whole categories. We cover those, and the Googlebot catch that comes with blocking "Training", in our guide to Cloudflare AI Crawl Control. This guide is about the rules layer underneath.

Two fields do most of the work in the rules below:

• cf.client.bot: true when the request comes from a known good bot that Cloudflare has verified by IP range, reverse DNS or cryptographic signature. It gives the same information as cf.bot_management.verified_bot.
• cf.verified_bot_category: the category of that verified bot. Values relevant here are "Search Engine Crawler" (Googlebot, Bingbot), "AI Search" (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot), "AI Assistant" (ChatGPT-User, Claude-User, Perplexity-User) and "AI Crawler" (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot).

Step 1: Find out what's being blocked today

Go to Security → Analytics (or Security → Events) and filter by user agent for OAI-SearchBot, PerplexityBot, Claude-SearchBot and ChatGPT-User. For each blocked or challenged request, note which service acted: a custom rule, managed rules, rate limiting, or Bot Fight Mode / Super Bot Fight Mode. Then open AI Crawl Control → Metrics and filter by status code. A run of 403 responses to AI search crawlers tells you which layer to fix first.

Step 2: Set crawler actions in AI Crawl Control

In AI Crawl Control → Crawlers, set the AI search and assistant crawlers to Allow. Block only the crawlers you've decided against. Cloudflare turns your blocks into a single WAF rule named "AI Crawl Control". You can edit that rule directly, for example to exempt certain paths, but edits made in the WAF aren't shown back in the AI Crawl Control dashboard, so keep a note of any changes.

Step 3: Deal with Bot Fight Mode (Free plan)

Bot Fight Mode is Cloudflare's free bot protection. It challenges traffic that matches known bot patterns. Cloudflare's documentation is explicit: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." Skip, Bypass and Allow actions have no effect on it.

So if Step 1 showed Bot Fight Mode challenging AI search crawlers, you have two options: turn Bot Fight Mode off (Security → Settings), or upgrade to a plan with Super Bot Fight Mode, where you can allow verified bots and use Skip rules. For most B2B marketing sites, turning it off and relying on the targeted rules below works well.

On Pro and higher plans, open Super Bot Fight Mode and set Verified bots to Allow.

Step 4: Let verified search crawlers skip bot protection and rate limits

Create a custom rule under Security → Security rules → Create rule → Custom rule and put it first in the list. In the expression editor, paste:

Rule 1: verified search and AI search crawlers, action Skip

(cf.client.bot and cf.verified_bot_category in {"Search Engine Crawler" "AI Search" "AI Assistant"})

Set the action to Skip and choose All Super Bot Fight Mode rules and All rate limiting rules. Leave managed rules (the WAF's attack protection) on: verified crawlers don't need to skip them, and skipping them only widens your attack surface. This rule makes sure Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user fetchers aren't challenged or throttled by mistake.

Skip doesn't work for Bot Fight Mode (see Step 3), so this rule matters on Pro plans and above.

Step 5: Block bots that pretend to be AI crawlers

Anyone can put "GPTBot" in a user agent, and Common Crawl warns that fake CCBots are common. Real AI crawlers are verified by Cloudflare, so you can safely block requests that use their names but aren't verified:

Rule 2: impostor AI crawlers, action Block

(
  lower(http.user_agent) contains "gptbot" or
  lower(http.user_agent) contains "oai-searchbot" or
  lower(http.user_agent) contains "chatgpt-user" or
  lower(http.user_agent) contains "claudebot" or
  lower(http.user_agent) contains "claude-searchbot" or
  lower(http.user_agent) contains "claude-user" or
  lower(http.user_agent) contains "perplexitybot" or
  lower(http.user_agent) contains "perplexity-user" or
  lower(http.user_agent) contains "ccbot"
) and not cf.client.bot

Set the action to Block and place it after Rule 1. Before turning it on, check Security Events for a few days to confirm that real crawler traffic shows as verified on your plan. If you see legitimate requests matching, use Managed Challenge instead while you investigate.

Step 6: Block training crawlers only where you need to

If you want AI training kept out of specific sections, such as a research library or customer docs, without affecting the rest of the site:

Rule 3: training crawlers on protected folders, action Block

(cf.verified_bot_category eq "AI Crawler" and (
  starts_with(http.request.uri.path, "/research/") or
  starts_with(http.request.uri.path, "/customer-docs/")
))

Pair this with matching Disallow lines in robots.txt (see Template 4) so well-behaved crawlers don't request those URLs in the first place. Blocking training this way is safer than the account-wide "Training" setting, which Cloudflare says can also block multi-purpose crawlers such as Googlebot, BingBot and Applebot.

Step 7: Rate-limit the unverified scrapers

Aggressive scrapers that don't identify themselves are the real load problem. Under Security rules → Rate limiting rules, create a rule that targets unverified automated traffic and leaves people and verified bots alone. For example, match not cf.client.bot on paths that scrapers hammer (search pages, product listings, faceted filters), with a threshold far above what any human would reach. Start with Managed Challenge rather than Block.

If a verified crawler is overloading your server, slow it down instead of blocking it. Anthropic supports Crawl-delay in robots.txt, and you can also contact the crawler's operator. Their documentation pages list contact details.

Step 8: Verify and keep an eye on it

• Security Events: filter by each of your rules for a week and check that nothing legitimate is being caught.
• AI Crawl Control → Metrics: AI search crawlers should show 200 responses, and the referral analytics should start showing visits from AI platforms.
• Directives tab: check which crawlers ignore your robots.txt. Those are candidates for a WAF block.
• Outside view: run the free LLM Readiness Check to confirm robots.txt is clean, then track AI referrals in GA4.

The full rule set at a glance

Frequently asked questions

Free tool

Check your robots.txt before you touch the firewall

The free LLM Readiness Check confirms that robots.txt allows the 15 major AI crawlers, so you know any remaining blocks are coming from your firewall.

Run the free check