AI Crawler & robots.txt Checker: Test Search, Training, and User Fetchers

robots.txt is a public crawler-policy file. It is not authentication, authorization, a privacy control, or a visibility guarantee. Use it to express crawl preferences for user-agent tokens documented by the relevant vendor, then verify server behavior and logs.

Crawler names and product effects can change. Check current official documentation before copying a rule.

What the live checker tests

The checker requests only five fixed public HTTPS locations on the domain you enter: the homepage, robots.txt, sitemap.xml, llms.txt, and agent.json. It evaluates the site root against current documented crawler tokens and shows Allowed, Blocked, or Not declared. A missing file and a failed network check remain different states.

It does not automatically change or modify your website. The generated policy block is a draft. Review it before publishing, and check it against your search, content-rights, legal, security, and infrastructure decisions.

Supporting files are not scored. llms.txt is an optional community-proposed aid, while agent.json is an experimental convention. Neither replaces accessible HTML, a sitemap, normal robots controls, authentication, or relevant structured data.

Search, training, and user-triggered access are separate choices

Current first-party documentation distinguishes multiple purposes:

• Search crawlers: OAI-SearchBot, Claude-SearchBot, and PerplexityBot support the vendors' search or answer retrieval. Googlebot controls Google Search crawling, including Search AI features.
• Training controls: GPTBot and ClaudeBot cover vendor-described model-training collection. Google-Extended is a separate control for specified Gemini and Vertex AI training or grounding uses outside Google Search.
• User-triggered fetchers: ChatGPT-User, Claude-User, and Perplexity-User can retrieve content after a user action. The vendors' current robots behavior differs, so a parsed rule is not proof that every user-triggered request will follow it.

Official definitions reviewed on 2026-08-18:

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Google common crawler documentation
• Google Search AI features guidance

Allowing access does not guarantee crawling, inclusion, ranking, citation, recommendation, or referral traffic. A published policy also does not prove that a request came from a genuine vendor bot; verify supported IP ranges or reverse DNS and inspect server logs where available.

Start with a content policy

Classify paths before editing the file:

Never rely on robots.txt to protect secrets. A disallowed URL may still be known, linked, logged, or requested by a client that ignores the file.

Understand user-agent scope

A robots group applies to the matching user-agent token. Different vendors may operate separate crawlers for search, model development, or user-triggered retrieval. Do not assume one token controls every product from a company.

User-agent: ExampleBot
Disallow: /private-preview/
Allow: /public-docs/

This example is only syntax. Replace ExampleBot with a currently documented token and verify the vendor's stated product scope.

Googlebot and Google-Extended are different controls

Google documents Googlebot as the crawler used for Google Search. Google-Extended is a standalone product token for specified Gemini and Vertex AI uses.

Google-Extended does not control or affect Google Search AI Overviews or AI Mode. Google also says it does not affect inclusion or ranking in Google Search. Search crawling and indexing remain governed by Google Search controls such as Googlebot and page-level indexing directives.

Review Google's current primary documentation:

• Google crawler overview
• Google common crawler tokens
• AI features and your website

Do not claim that allowing Google-Extended enables AI Overviews, or that blocking it removes a site from AI Overviews or AI Mode.

Verify vendor-specific bots

For each token:

1. Open the vendor's current official crawler documentation.
2. Record the token, product purpose, and review date.
3. Decide whether the use aligns with your content policy.
4. Add the narrowest practical rule.
5. Validate the live robots.txt.
6. Inspect logs where available.
7. Re-review after policy or product changes.

Anthropic, for example, publishes a current help page describing its bot tokens and robots behavior. Use the Anthropic crawler documentation rather than a copied third-party list.

robots.txt, noindex, and access control

• robots.txt expresses crawl preferences.
• a supported noindex directive controls indexing only when the crawler can read it;
• authentication and authorization protect private resources;
• a firewall or WAF may control network access.

Blocking a crawler from a page can prevent it from seeing a noindex meta tag on that page. Design crawl and indexing controls together.

Monitoring checklist

• live file returns the intended content and status;
• rules apply to the expected host and subdomain;
• important CSS, JavaScript, image, and page resources are not blocked accidentally;
• private routes require authentication;
• logs are reviewed for load and policy violations;
• official crawler documentation and internal decisions have review dates.

Allowing a crawler does not guarantee crawling, indexing, citation, ranking, or referral traffic. Blocking it does not erase information already obtained through other lawful sources or earlier crawls.

What to do after the checker returns a result

Use the result as a starting point for a specific check, not as a verdict on whether an AI product recommends your site:

• Allowed: review an important public page as well as the root. A rule that allows / may still restrict a deeper path; a CDN or login requirement may also prevent access.
• Blocked: identify the matching group and path rule. Confirm that the restriction is unintended before changing it; a training restriction may be a deliberate policy choice.
• Not declared: inspect the full file and the crawler's documented behavior. Do not add an Allow rule merely to turn the label green.
• Missing file or failed request: distinguish an absent policy from a network, server, or firewall problem. Recheck the actual response before choosing a remedy.

The live checker evaluates the site root, not every URL. Keep a short manual test list: homepage, one product page, one documentation page, and a private route that must still require authentication. Do not submit private credentials to a crawler checker.

Before-and-after verification for one policy change

Save the current file and record the rule you intend to change. Review which user-agent group applies: Google's robots.txt specification explains that a specific matching group is not combined with the wildcard group. Do not assume rules under User-agent: * automatically carry over to a separately named crawler.

After publishing a reviewed change:

1. Open the live /robots.txt on the intended host and confirm that the response contains the new plain-text rules, not an error or login page.
2. Recheck the intended public paths and any separate host or subdomain in scope.
3. Verify that private routes still require authentication; a robots rule is not that protection.
4. Check available server logs for real requests and unexpected errors. A local request with a bot-like user-agent does not prove a vendor can fetch the page.
5. If access changed unexpectedly, restore the saved policy and verify the live response again.

This confirms the policy and access behavior you tested, not indexing or future citations. For broader page findings, run an AI visibility audit; review the sample report first if you want to see which evidence it retains.

FAQ schema boundary

FAQ schema does not guarantee an AI Overview or AI citation. It is separate from crawler access and must match visible, eligible FAQ content under current search documentation.

Frequently Asked Questions

Does blocking Google-Extended remove my site from AI Overviews or AI Mode?

No. Google explicitly says Google-Extended does not control those Google Search features.

Can robots.txt protect private pages?

No. Use authentication and authorization. A robots rule is public and advisory.

Should I allow every AI crawler?

There is no universal answer. Decide from current vendor documentation, content rights, server cost, legal obligations, and business policy.

How often should I review crawler rules?

Review after material policy, vendor, product, infrastructure, or content changes. A calendar review is useful, but no fixed cadence fits every site.

Does allowing a crawler improve AI visibility?

No improvement is guaranteed. Access is only one possible prerequisite for a particular crawler's documented use.