Reflyma / AI Bot Access Checker

Which AI crawlers can actually reach your site?

One scan across every major AI bot — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and more. Know exactly who you're allowing in.

Free to run
No account required
Live results
Evidence-backed

AI Bot Access Checker

Enter a URL above to run this tool

Ready to run

yourbrand.com · ai bot access checkerready to run

Enter a URL above to run this tool.

How it works

From URL to evidence-backed result.

Fetch robots.txt
Follow redirects and read every directive.
Match against AI bots
Test 12+ AI user agents against your rules.

What you get

Built for AI search visibility.

Every major AI bot

Tracks OpenAI, Anthropic, Google, Perplexity, Microsoft, Common Crawl and more — kept up to date.

Wildcard-aware

Correctly handles User-agent: * fallbacks, precedence and case sensitivity.

Copy-paste fixes

Turn a policy decision into a robots.txt snippet you can ship in one line.

Field guide

How AI crawler access actually works

Every AI answer engine reaches your site through a named user agent, and every one of them can be blocked — deliberately in robots.txt, or accidentally by a CDN rule, a firewall, or a bot-management product enabled by default. This checker tests each known agent against your live rules and shows who is allowed in.

Which crawlers we test

We evaluate the published user agents for the major answer engines and their training crawlers separately, because blocking one does not imply blocking the other. GPTBot, OAI-SearchBot and ChatGPT-User behave differently; so do Google-Extended and Googlebot.

For each agent we apply your robots.txt rules the way a compliant crawler would: the most specific matching group wins, and an unmatched agent falls back to the wildcard group.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
Google: Googlebot, Google-Extended
Anthropic: ClaudeBot, anthropic-ai
Perplexity: PerplexityBot
Common Crawl: CCBot, plus Applebot and Bingbot

Blocked on purpose versus by accident

Blocking training crawlers while allowing search crawlers is a legitimate policy choice, and plenty of publishers make it. The problem is the accidental version: a copy-pasted robots.txt, an edge rule that challenges unknown agents, or a default bot-protection setting that returns 403 to everything without a browser fingerprint.

The distinction matters because a blocked retrieval crawler means your pages cannot be cited at all, no matter how good the content is.

What to do with the results

Start with retrieval agents — the ones that fetch pages at answer time. If any of those are denied and you want AI visibility, fix robots.txt first, then verify at the edge that real requests are not being challenged.

Remember robots.txt is advisory. A rule that allows a crawler does not help if your CDN returns a challenge page; test from outside your network to confirm.

Allow retrieval agents you want citations from
Decide training-crawler policy deliberately
Check CDN and WAF bot rules, not only robots.txt
Avoid blanket wildcard disallow rules
Re-test after every infrastructure change

Access is necessary, not sufficient

Open access only puts you in the running. Whether you are then quoted depends on whether the page is server-rendered, states its answer plainly, and is more accurate than the alternatives the model retrieved.

Run the robots.txt analyzer and answer preview alongside this check to see the full picture.

FAQ

Questions, answered.

Should I block AI training crawlers?+

That is a business decision. Blocking training crawlers protects content reuse; blocking retrieval crawlers removes you from AI answers entirely. Most brands allow retrieval and decide on training separately.

Do all crawlers obey robots.txt?+

The major named agents documented by their vendors do. Unidentified scrapers often do not, which is what firewall rules are for.

Why does the checker say allowed when my logs show blocks?+

robots.txt permission is separate from network-level blocking. Check your CDN or WAF bot-management settings.

How often should I re-check?+

After every robots.txt or infrastructure change, and monthly otherwise — vendors add new agents regularly.

Should I block AI bots?+

Depends on strategy. Blocking means no AI answers cite you. Allowing means visibility with less traffic. Reflyma helps you measure both.

Are these user agents real?+

Yes — pulled from the vendors' official docs and updated as they change.

More free tools

Keep exploring.

All tools

Reflyma

Track your brand across ChatGPT, Gemini, Perplexity and Claude.

Start free See pricing