# AGI Search Labs — robots.txt # We build AI-visibility strategy for a living, so this file practices what # we sell: standard search crawlers and major AI/LLM crawlers are explicitly # welcomed, not blocked. # # Note: the one page we don't want indexed (our paid-traffic landing page) # is excluded via an on-page tag # instead of a Disallow rule here, so crawlers can still fetch it, read the # noindex directive, and drop it cleanly — a Disallow would hide that tag # from them and risk the opposite of the intended effect. User-agent: * Allow: / # --- Major AI / LLM crawlers (explicitly allowed) --- # Includes all 15 crawlers the LLM Readiness Check tests. User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Applebot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: meta-externalagent Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / User-agent: Bytespider Allow: / User-agent: cohere-ai Allow: / Sitemap: https://www.agisearchlabs.com/sitemap.xml