AI Crawler & robots.txt Checker: Test Search, User, and Training Bots

Test any public domain against current OpenAI, Anthropic, Perplexity, and Google crawler tokens. Separate search bots, user fetchers, and training controls.
Nov 18, 2026

Use this live AI crawler & robots.txt checker to inspect how your published root robots.txt treats documented search crawlers, model-training crawlers, and user-triggered fetchers—plus whether your homepage, sitemap.xml, optional llms.txt, and experimental agent.json respond over HTTPS.

Free live checker

Check robots.txt policies for current AI crawlers

Review the published root policy, supporting access files, and dated vendor definitions. This does not verify real bot identity, visits, indexing, citations, or training use.

The checker makes bounded requests only to the public HTTPS homepage, robots.txt, sitemap.xml, llms.txt, and agent.json on the entered host. It does not automatically change or modify your site.

Reference matrix: search crawlers vs training controls vs user fetchers

Blocking all AI user-agent tokens indiscriminately can accidentally block search and citation retrieval while leaving private paths unprotected. Current vendor documentation separates crawler tokens by purpose (official docs reviewed on 2026-08-18; always re-check vendor pages before editing production rules):

User-agent tokenVendorDocumented purposeKey operational boundary
OAI-SearchBotOpenAISearch and answer source discoverySeparate from GPTBot; blocking it restricts search discovery in OpenAI search features
GPTBotOpenAIModel-training collectionControls OpenAI training crawls without needing to block OAI-SearchBot
ChatGPT-UserOpenAIUser-initiated page fetchesTriggered by user actions; robots handling can differ from automated background crawls
Claude-SearchBotAnthropicSearch retrievalSeparate from ClaudeBot training collection
ClaudeBotAnthropicModel-training collectionDocumented token for opting out of Anthropic model-training crawls
Claude-UserAnthropicUser-initiated fetchesTriggered when a Claude user requests a specific page
PerplexityBotPerplexitySearch index and answer retrievalDocumented search crawler for Perplexity answer citations
Perplexity-UserPerplexityUser-initiated fetchesDocumented separately from background PerplexityBot indexing
GooglebotGoogleGoogle Search crawling (including AI Overviews and AI Mode)Blocking Googlebot removes pages from Google Search broadly
Google-ExtendedGoogleSpecified Gemini and Vertex AI training/grounding outside Google SearchDoes not control Google Search rankings, AI Overviews, or AI Mode

For detailed vendor links and step-by-step robots.txt rule syntax, read our companion AI crawler & robots.txt policy guide.

Three common robots.txt mistakes that hurt search and AI discovery

  1. Blocking noindex pages in robots.txt: If a consolidated or low-value URL carries <meta name="robots" content="noindex, follow">, crawlers must be able to fetch the page to see the noindex directive. Disallowing that path in robots.txt hides the directive rather than de-indexing the URL cleanly.
  2. Confusing Google-Extended with Google Search AI Overviews: Google documents that Googlebot and standard page-level indexing controls govern Google Search AI features, whereas Google-Extended applies to specified non-Search Gemini and Vertex AI uses.
  3. Assuming robots.txt overrides CDN/WAF blocks or protects secrets: A User-agent: * / Allow: / file does not help if a CDN firewall challenges automated HTTP requests with HTTP 403, and Disallow: never replaces authentication on private /dashboard or /admin routes.

What to check after your robots.txt passes

Passing a robots.txt check confirms crawl preferences are declared—it does not guarantee crawling, indexing, citations, or recommendations. Next:

Frequently Asked Questions

Does allowing AI search crawlers guarantee ChatGPT or Perplexity citations?

No. Allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot removes a self-inflicted crawl block, but each system still decides independently what to crawl, index, rank, and cite.

Can I block AI model training while still allowing AI search citations?

Vendors that publish separate tokens—such as OpenAI (OAI-SearchBot vs GPTBot), Anthropic (Claude-SearchBot vs ClaudeBot), and Google (Googlebot vs Google-Extended)—document distinct controls for search and training. Check each vendor's current documentation to confirm exact scope.

Why does the checker show "Not declared" instead of "Allowed"?

"Not declared" means your robots.txt had no matching User-agent group (including *) for that token. Standard robots parsers treat undeclared paths as allowed by default, but the checker reports the literal file state rather than assuming explicit intent.

Does this checker verify real bot traffic in my server logs?

No. It inspects your public HTTPS robots.txt and supporting files at a point in time. Anyone can spoof a User-Agent header, so verify genuine crawler visits using vendor-published IP ranges or reverse DNS against your server or CDN logs.

Next Step

Run your own AI visibility audit

Use what you learned here, then check your own site for weak positioning, missing comparison pages, thin FAQs, and other answer-readiness gaps.

Related Resources

Keep exploring the pages most closely connected to this topic.