Robots.txt & Sitemap Viewer
robots.txt
sitemap.xml
GuideA Practical Technical SEO Check for a Web Page
Read guide →
Review links, robots directives, sitemaps, screenshots, and repeated page terms without treating one score as a ranking guarantee.
Frequently asked questions
What does robots.txt control?
It gives participating crawlers path-level crawl instructions. It is not an access-control mechanism and does not guarantee deindexing.
What belongs in an XML sitemap?
Include canonical, indexable URLs that return successful responses. Exclude private endpoints, redirects, and error pages.
Why might a robots or sitemap fetch be empty?
The file may not exist, may return an error, may block automated requests, or may be larger than the fetch limit.
Can a sitemap guarantee indexing?
No. It helps search engines discover URLs, but each engine decides whether and when to crawl and index them.