A sitemap AI crawlers can actually use.

What the analyzer inspects

We locate your sitemap from robots.txt and the conventional /sitemap.xml path, expand sitemap index files, and parse every child document. Each URL is checked for a valid absolute form, correct protocol and host, and a well-formed lastmod date.

We also compare the sitemap against what robots.txt allows, because listing a URL you have disallowed is a direct contradiction that crawlers resolve by ignoring the page.

• Sitemap discovery via robots.txt and common paths
• Valid XML structure and namespace
• Index files expanded to their children
• Absolute URLs on the canonical host
• lastmod format, entry counts and size limits

Common problems and their cost

Stale lastmod values are the most damaging: when every entry claims today's date, crawlers stop trusting the field and recrawl on their own schedule, which is usually slower than you want. Entries pointing at redirects or 404s have the same effect on a smaller scale.

Oversized files are next. A single sitemap should stay under fifty thousand URLs and fifty megabytes uncompressed; beyond that, split it into an index with topical children so you can see per-section health.

• Honest lastmod dates that reflect real edits
• No redirected, dead or disallowed URLs
• One canonical host and protocol throughout
• Split large sitemaps into a clean index
• Reference the sitemap from robots.txt

Why this matters for AI answers

Retrieval systems rely on the same crawl infrastructure as classic search. If a page is never fetched, it cannot be embedded, retrieved, or cited — the quality of the writing is irrelevant.

A tidy sitemap also makes your own diagnostics easier: when coverage drops, you can tell whether the page is missing, blocked, or simply not selected.

A maintenance routine that works

Generate the sitemap from your content system rather than by hand, so new pages appear automatically and removed pages disappear. Re-run this analyzer after each release, and keep an eye on the total entry count as a cheap change detector.

If the count moves sharply without a corresponding release, something in your build or CMS changed unexpectedly.