Robots.txt Analyzer
Read your robots.txt like a search engine does.
We parse every directive, spot silent conflicts and translate technical rules into plain English — with AI-crawler awareness built in.
- Free to run
- No account required
- Live results
- Evidence-backed
Robots.txt Analyzer
Enter a URL above to run a live scan
yourbrand.com · robots.txt analyzerwaiting
Enter a URL above to run a live scan.
How it works
From URL to evidence-backed result.
- Fetch robots.txt
Follow redirects and validate delivery.
- Parse directives
Group by user agent, resolve precedence.
- Flag issues
Warn on ambiguous, conflicting or dangerous rules.
What you get
Built for AI search visibility.
Silent-issue detection
Catches mistakes that leave AI bots blocked without you knowing.
AI-crawler aware
Special reporting for GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Safe recommendations
Every suggestion is scoped so it can't accidentally deindex your site.
FAQ
Questions, answered.
Does Disallow block AI training?+
For most vendors, yes — but not all. We flag per-bot behaviour so you don't rely on assumptions.
Where should robots.txt live?+
At the root: /robots.txt on your primary domain, served over HTTPS with a 200 response.
Reflyma