Use the analyzer to turn robots.txt directives into a readable audit before making crawler-access changes.
How to use it
1
Enter your website root URL.
2
Run the analyzer.
3
Review User-agent, Disallow, Allow and sitemap directives.
4
Identify rules that conflict with your intended crawl policy.
5
Edit robots.txt, deploy it at the domain root and rerun the analysis.
Best practices
- A robots.txt file controls crawling, not whether a URL can exist in search results.
- Always test important paths after changing broad disallow rules.
What you get
A plain-English robots.txt audit with actionable crawler-policy findings.