Read your robots.txt like a search engine does.

How the file is parsed

Rules are grouped by user-agent. A crawler uses the most specific group that names it and ignores the others entirely — it does not merge a specific group with the wildcard group, which is the source of most surprises.

Within a group, the longest matching path wins, and Allow beats Disallow at equal length. We apply those rules literally so you see the outcome rather than the intention.

• Group structure and user-agent matching
• Allow and Disallow precedence by path length
• Sitemap declarations and their absolute URLs
• Crawl-delay and unsupported directives
• Syntax errors, encoding issues and stray HTML

The failures that cost the most

A production file still carrying 'Disallow: /' from a staging deployment is the classic catastrophic case. Less obvious, and far more common, is a specific AI user-agent group that allows nothing because its author assumed the wildcard rules still applied.

Blocking asset directories is the quiet one: when CSS or JavaScript is disallowed, renderers cannot lay the page out and may misjudge what the page contains.

• Never ship 'Disallow: /' to production
• Repeat needed Allow rules inside specific agent groups
• Keep CSS, JS and image paths crawlable
• Declare your sitemap with an absolute URL
• Serve the file as plain text with a 200 status

What robots.txt cannot do

It is not a security control and not a reliable way to keep a page out of an index. Disallowed URLs can still be listed if other sites link to them; use noindex or authentication for anything that must stay private.

It is also advisory. Well-behaved crawlers comply, unnamed scrapers do not, and blocking those is a job for your firewall.

Keeping it correct over time

Treat robots.txt as code: keep it in version control, review changes in pull requests, and re-run this analyzer after each deploy. Most damaging robots.txt incidents are deploys, not decisions.

Pair the check with the AI bot access checker to confirm the specific answer engines you care about can still reach your pages.