Your Firewall May Be Hiding You From AI: 6 Places Bots Get Blocked

In this article

1. Why a clean robots.txt isn't enough
2. 1. CDN bot settings
3. 2. Your hosting platform's firewall
4. 3. WordPress security and "anti-scraping" plugins
5. 4. Geo-blocking and data-center IP blocks
6. 5. Rate limits, challenge pages and error responses
7. 6. Leftover noindex, passwords and directives
8. A 15-minute diagnosis
9. Frequently asked questions

Why a clean robots.txt isn't enough

robots.txt is a request crawlers choose to honor. Firewalls are enforcement. A crawler that robots.txt allows can still be refused by anything that sits between it and your page: a CDN, a hosting platform's firewall, a security plugin, or your own server configuration. OpenAI's help center says it plainly: to be eligible for ChatGPT search, a site must allow OAI-SearchBot, and the host or CDN must allow traffic from OpenAI's published IP addresses.

These blocks are usually invisible to the marketing team. Nobody decided to hide the company from AI. Someone turned on a security setting, and the side effect only shows up as a gap in AI answers months later. Here are the six places to look, in the order we check them.

1. CDN bot settings

CDNs such as Cloudflare, Akamai and Fastly see every request first, and all of them offer bot management that can classify and block AI crawlers. Cloudflare has the most settings that affect AI crawlers:

• AI Crawl Control, where each AI crawler can be set to Allow or Block.
• The Search, Agent and Training settings added in July 2026. From September 15, 2026, new domains block Training and Agent bots on pages with ads by default. Cloudflare also says that customers who block Training will block multi-purpose crawlers such as Googlebot, Applebot and BingBot.
• The older Block AI bots toggle and Bot Fight Mode, which challenges automated traffic and can't be overridden with firewall rules.

How to check: review the CDN's security event log for requests from AI crawler user agents that were blocked or challenged. On Cloudflare, follow our AI Crawl Control guide and WAF rules guide.

2. Your hosting platform's firewall

Modern hosting platforms have their own firewalls, separate from any CDN. Vercel is a good example of what to look for. Its firewall has two managed rulesets that affect AI crawlers, both off by default:

• AI Bots Managed Ruleset: set to Deny, it "blocks all traffic identified as coming from AI bots".
• Bot Protection Managed Ruleset: set to Challenge, it serves a JavaScript challenge to traffic that is unlikely to be a browser. Crawlers that don't run JavaScript can't pass it.

Managed WordPress hosts and other platforms often have similar bot controls, sometimes switched on by the host during a traffic spike or attack and never switched off.

How to check: open your host's firewall or security settings and look for anything mentioning bots, AI or challenges. On Vercel, set the AI Bots ruleset to Log first to see what it would catch. If you need Bot Protection on, add a bypass rule for verified AI search crawlers.

3. WordPress security and "anti-scraping" plugins

WordPress security plugins protect against real attacks, but several of their common features also catch crawlers:

• Rate limiting that blocks any client making many requests per minute, which is exactly what a crawler does.
• "Block fake crawlers" options that only recognize Google and Bing as legitimate and treat every other bot as suspicious.
• Country blocking (see point 4).
• Dedicated AI-blocking plugins or toggles that add AI crawlers to a blocklist or to robots.txt.

How to check: review the plugin's block log for AI crawler names, and look in its settings for rate limits, crawler rules and country rules. Then make sure robots.txt, as served live, is still the file you expect. Some plugins and SEO tools generate it dynamically.

4. Geo-blocking and data-center IP blocks

Blocking traffic by country, by network (ASN) or by cloud provider is a common way to cut spam and attacks. The catch: AI crawlers are data-center traffic. OpenAI's and Anthropic's published crawler ranges include cloud-provider addresses. A rule such as "block all hosting and cloud networks" or "allow only our sales countries" can shut out every AI crawler at once.

Blocking IPs has a second side effect. Anthropic advises against IP blocking specifically because it stops its crawlers from reading your robots.txt, so your crawl preferences can't be applied.

How to check: list every country, ASN and IP-range rule in your CDN, host and server configuration. Exempt verified crawlers before any broad block. Most CDNs can match "verified bot" as a condition.

5. Rate limits, challenge pages and error responses

Some blocks don't look like blocks. A crawler that gets a CAPTCHA page, a "checking your browser" interstitial or a 429 Too Many Requests response simply leaves without your content. Worse, a challenge page served with a 200 status can be indexed in place of the real page.

Server errors matter too. If /robots.txt returns a 5xx error, RFC 9309 tells crawlers to assume the whole site is off-limits until it recovers.

How to check: in your server logs, filter AI crawler requests by status code. Any 403, 429 or 503 responses to verified crawlers point to a firewall or capacity problem. Our server log guide shows how to separate real crawlers from fakes first.

6. Leftover noindex, passwords and directives

Finally, the blocks that were meant to be temporary:

• X-Robots-Tag: noindex HTTP headers or <meta name="robots" content="noindex"> tags copied from a staging site. They keep pages out of Google and Bing, and with them Google AI Overviews, AI Mode and Copilot.
• Password protection or IP allowlists left on sections after a redesign.
• Bing's NOARCHIVE and NOCACHE. Bing's February 2026 guidelines say NOARCHIVE keeps content out of Copilot answers and grounding, and NOCACHE limits Copilot to the URL, title and snippet.
• A launch-day Disallow: / in robots.txt that was never removed.

How to check: fetch your key pages and read the response headers:

curl -sI https://www.example.com/services | grep -iE "^(HTTP|x-robots-tag)"

Then check the page source for robots meta tags. The free LLM Readiness Check flags a noindex on your homepage automatically.

A 15-minute diagnosis

1. Run the LLM Readiness Check to rule out robots.txt, noindex and JavaScript problems.
2. Fetch a key page with an AI crawler's user agent and confirm a 200 status with real content:

   curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.example.com/

   This tests user-agent rules only. IP-based rules won't trigger from your machine.
3. Search your CDN or host firewall log for OAI-SearchBot, PerplexityBot, Claude-SearchBot and ChatGPT-User.
4. Check server logs for status codes returned to verified AI crawlers.
5. List every geo, ASN, rate-limit and plugin rule, and make sure verified crawlers are exempt.

If AI crawlers never appear in your logs at all, the block is almost certainly upstream, at the CDN or host.

Frequently asked questions

Free tool

Start with the layer you can test in 10 seconds

The free LLM Readiness Check confirms your robots.txt, indexability and raw HTML are AI-ready, so you can focus on the firewall layers below.

Run the free check