Use this checker when you need a quick view of crawler permissions before changing robots.txt.
How to use it
1
Enter your website root URL.
2
Run the bot access check.
3
Review each supported AI user agent and its allow or block status.
4
Compare the result with your intended crawler policy.
5
Update robots.txt only when the current access rules do not match your policy.
Best practices
- Be deliberate about exclusions: blocking a crawler can reduce discovery or retrieval opportunities.
- Review both broad User-agent rules and specific overrides.
What you get
A clear allow/block view for supported AI crawlers.