How to Verify Real AI Bot Traffic in Your Server Logs (and Spot Fakes)

In this article

1. Why verifying AI bots matters
2. Step 1: Find AI crawlers in your logs
3. Step 2: Know how each crawler can be verified
4. Step 3: Verify a whole log file with a script
5. Step 4: Spot-check with DNS
6. How to spot fake AI crawlers
7. What verified AI traffic tells you
8. Frequently asked questions

Why verifying AI bots matters

Your server logs are the only first-hand record of which AI crawlers actually read your site. They answer questions no SEO tool can: is OAI-SearchBot getting your new pricing page, are AI crawlers hitting errors, and are the bots that claim to be GPTBot really OpenAI?

That last question matters because a user agent is just text the client sends. Scrapers routinely borrow the names of well-known crawlers to get past basic filters. If you count them as real AI traffic, your reports are wrong. If you whitelist them by user agent, you've let a scraper in. Verification fixes both problems.

Step 1: Find AI crawlers in your logs

Most servers write access logs in the "combined" format, which includes the client IP, the request, the status code and the user agent. On Nginx they're usually in /var/log/nginx/access.log and on Apache in /var/log/apache2/access.log or /var/log/httpd/access_log. Behind a CDN, make sure the logged IP is the real client IP, not the CDN's, by checking that your server reads the forwarded IP header your CDN provides.

Count requests by AI crawler

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Extended|Googlebot|bingbot|Applebot|meta-externalagent|CCBot" access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

See which pages OAI-SearchBot requests, and the status codes it gets

grep "OAI-SearchBot" access.log | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -30

Google-Extended and Applebot-Extended won't appear, because they're robots.txt tokens, not crawlers. If you see none of the AI search crawlers at all over a few weeks, they may be blocked before they reach your server. See where AI crawlers get blocked.

Step 2: Know how each crawler can be verified

Google also publishes separate lists for its user-triggered fetchers and agents (for example user-triggered-agents.json) in the same folder as common-crawlers.json. Most of these JSON files share one simple format: a prefixes array of objects with an ipv4Prefix or ipv6Prefix in CIDR notation.

Step 3: Verify a whole log file with a script

This Python 3 script downloads the published lists, reads your access log, and reports each AI crawler's requests as verified or fake. It uses only the standard library.

verify_ai_bots.py

import ipaddress, json, re, sys, urllib.request
from collections import Counter

LISTS = {
    "GPTBot":          "https://openai.com/gptbot.json",
    "OAI-SearchBot":   "https://openai.com/searchbot.json",
    "ChatGPT-User":    "https://openai.com/chatgpt-user.json",
    "ClaudeBot":       "https://claude.com/crawling/bots.json",
    "Claude-SearchBot":"https://claude.com/crawling/bots.json",
    "Claude-User":     "https://claude.com/crawling/bots.json",
    "PerplexityBot":   "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
    "Googlebot":       "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "bingbot":         "https://www.bing.com/toolbox/bingbot.json",
    "Applebot":        "https://search.developer.apple.com/applebot.json",
    "CCBot":           "https://index.commoncrawl.org/ccbot.json",
}

def load(url, cache={}):
    if url not in cache:
        data = json.load(urllib.request.urlopen(url, timeout=20))
        cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
                      for p in data.get("prefixes", [])
                      if p.get("ipv4Prefix") or p.get("ipv6Prefix")]
    return cache[url]

line_re = re.compile(r'^(\S+) .*"([^"]*)"$')   # combined log format: IP first, UA last
results = Counter()
for line in open(sys.argv[1], errors="ignore"):
    m = line_re.match(line.strip())
    if not m:
        continue
    ip, ua = m.groups()
    for bot, url in LISTS.items():
        if bot.lower() in ua.lower():
            try:
                addr = ipaddress.ip_address(ip)
                ok = any(addr in net for net in load(url))
            except ValueError:
                ok = False
            results[(bot, "verified" if ok else "FAKE")] += 1
            break

for (bot, status), n in sorted(results.items()):
    print(f"{bot:18} {status:9} {n}")

Run it with python3 verify_ai_bots.py /var/log/nginx/access.log. A clean result shows mostly "verified" lines. A large "FAKE" count for one crawler usually means one scraper using that name. List its IPs with grep and block them.

Companies occasionally change a file's location or format. If every request for one crawler comes back "FAKE", open that JSON file in a browser and check the URL and the field names before drawing conclusions.

Step 4: Spot-check with DNS

For Google, Bing, Apple and Common Crawl you can verify a single IP from the command line. Look up the hostname, check the domain, then confirm the hostname points back to the same IP:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Both steps matter. Anyone who controls an IP block can set its reverse DNS to say "googlebot.com", but they can't make Google's DNS point that name back to their IP.

How to spot fake AI crawlers

• The IP isn't on the published list and reverse DNS doesn't match. This is the definitive test.
• It requests things a crawler wouldn't, such as /wp-login.php, /.env, admin paths or search URLs with odd parameters.
• It ignores your robots.txt even though the real crawler obeys it.
• The request rate is far higher than the real crawler's usual pattern, or comes in bursts from many unrelated IPs.
• The user agent is malformed: an old version number, a typo, or the crawler name bolted onto a normal browser string in an unusual way.

Block impostors at the firewall rather than in robots.txt, because scrapers don't read robots.txt. On Cloudflare, one rule blocks every request that uses an AI crawler's name but isn't verified. It's Rule 2 in our Cloudflare WAF guide.

What verified AI traffic tells you

• OAI-SearchBot, Claude-SearchBot and PerplexityBot hits show which pages are being indexed for AI search. If your key service and product pages never appear, check internal links, your sitemap and whether those pages are readable without JavaScript.
• ChatGPT-User, Claude-User and Perplexity-User hits are live lookups during conversations. A spike on a page often means people are asking assistants about the topic, or about you.
• Status codes: 403, 429 or 503 responses to verified crawlers mean your firewall or server is turning them away.
• 404s: AI crawlers request a lot of missing URLs. In Vercel and MERJ's analysis, about 35% of ChatGPT's and Claude's crawler fetches hit 404 pages, against about 8% for Googlebot. Redirect old URLs that still matter.

For the visits AI platforms send to you, as opposed to the crawlers that read your site, see how to track AI referral traffic in GA4.

Frequently asked questions

Free tool

Logs look quiet? Check whether AI crawlers can get in

If AI crawlers never appear in your logs, something may be blocking them. The free LLM Readiness Check tests your robots.txt against 15 AI crawlers in 10 seconds.

Run the free check