AI Crawler & robots.txt Checker: Test Search, User, and Training Bots
Use this live AI crawler & robots.txt checker to inspect how your published root robots.txt treats documented search crawlers, model-training crawlers, and user-triggered fetchers—plus whether your homepage, sitemap.xml, optional llms.txt, and experimental agent.json respond over HTTPS.
Free live checker
Check robots.txt policies for current AI crawlers
Review the published root policy, supporting access files, and dated vendor definitions. This does not verify real bot identity, visits, indexing, citations, or training use.
The checker makes bounded requests only to the public HTTPS homepage, robots.txt, sitemap.xml, llms.txt, and agent.json on the entered host. It does not automatically change or modify your site.
Reference matrix: search crawlers vs training controls vs user fetchers
Blocking all AI user-agent tokens indiscriminately can accidentally block search and citation retrieval while leaving private paths unprotected. Current vendor documentation separates crawler tokens by purpose (official docs reviewed on 2026-08-18; always re-check vendor pages before editing production rules):
| User-agent token | Vendor | Documented purpose | Key operational boundary |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search and answer source discovery | Separate from GPTBot; blocking it restricts search discovery in OpenAI search features |
GPTBot | OpenAI | Model-training collection | Controls OpenAI training crawls without needing to block OAI-SearchBot |
ChatGPT-User | OpenAI | User-initiated page fetches | Triggered by user actions; robots handling can differ from automated background crawls |
Claude-SearchBot | Anthropic | Search retrieval | Separate from ClaudeBot training collection |
ClaudeBot | Anthropic | Model-training collection | Documented token for opting out of Anthropic model-training crawls |
Claude-User | Anthropic | User-initiated fetches | Triggered when a Claude user requests a specific page |
PerplexityBot | Perplexity | Search index and answer retrieval | Documented search crawler for Perplexity answer citations |
Perplexity-User | Perplexity | User-initiated fetches | Documented separately from background PerplexityBot indexing |
Googlebot | Google Search crawling (including AI Overviews and AI Mode) | Blocking Googlebot removes pages from Google Search broadly | |
Google-Extended | Specified Gemini and Vertex AI training/grounding outside Google Search | Does not control Google Search rankings, AI Overviews, or AI Mode |
For detailed vendor links and step-by-step robots.txt rule syntax, read our companion AI crawler & robots.txt policy guide.
Three common robots.txt mistakes that hurt search and AI discovery
- Blocking
noindexpages inrobots.txt: If a consolidated or low-value URL carries<meta name="robots" content="noindex, follow">, crawlers must be able to fetch the page to see thenoindexdirective. Disallowing that path inrobots.txthides the directive rather than de-indexing the URL cleanly. - Confusing
Google-Extendedwith Google Search AI Overviews: Google documents thatGooglebotand standard page-level indexing controls govern Google Search AI features, whereasGoogle-Extendedapplies to specified non-Search Gemini and Vertex AI uses. - Assuming
robots.txtoverrides CDN/WAF blocks or protects secrets: AUser-agent: * / Allow: /file does not help if a CDN firewall challenges automated HTTP requests with HTTP 403, andDisallow:never replaces authentication on private/dashboardor/adminroutes.
What to check after your robots.txt passes
Passing a robots.txt check confirms crawl preferences are declared—it does not guarantee crawling, indexing, citations, or recommendations. Next:
- Inspect actual brand answers: Run the AI visibility checker to see whether sampled GPT answers mention or cite your domain.
- Check live prompt citations: Test a category query in the ChatGPT visibility check.
- Review optional Markdown indexing: If you publish
/llms.txt, validate its links with the freellms.txtgenerator.
Frequently Asked Questions
Does allowing AI search crawlers guarantee ChatGPT or Perplexity citations?
No. Allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot removes a self-inflicted crawl block, but each system still decides independently what to crawl, index, rank, and cite.
Can I block AI model training while still allowing AI search citations?
Vendors that publish separate tokens—such as OpenAI (OAI-SearchBot vs GPTBot), Anthropic (Claude-SearchBot vs ClaudeBot), and Google (Googlebot vs Google-Extended)—document distinct controls for search and training. Check each vendor's current documentation to confirm exact scope.
Why does the checker show "Not declared" instead of "Allowed"?
"Not declared" means your robots.txt had no matching User-agent group (including *) for that token. Standard robots parsers treat undeclared paths as allowed by default, but the checker reports the literal file state rather than assuming explicit intent.
Does this checker verify real bot traffic in my server logs?
No. It inspects your public HTTPS robots.txt and supporting files at a point in time. Anyone can spoof a User-Agent header, so verify genuine crawler visits using vendor-published IP ranges or reverse DNS against your server or CDN logs.
Next Step
Run your own AI visibility audit
Use what you learned here, then check your own site for weak positioning, missing comparison pages, thin FAQs, and other answer-readiness gaps.
Related Resources
Keep exploring the pages most closely connected to this topic.
