How this works
- We read /robots.txt and parse the rules for all crawlers.
- We flag "Disallow: /" and blocked CSS or JavaScript folders.
- We find your XML sitemap from robots.txt or common locations and count the entries.
Robots.txt and Sitemap Checker
Make sure search engines are allowed to crawl your site and can find your sitemap.
Enter a domain only, no http:// needed.
Frequently asked questions
Why is my new site not showing in Google?
A staging setting that blocks crawlers sometimes goes live by mistake. Check robots.txt and the noindex meta tag.
Does Disallow remove a page from Google?
No. It only stops crawling. A blocked page can still appear in results without a description. Use noindex to remove pages.
Where do I submit a sitemap?
In Google Search Console under Sitemaps, and by adding a Sitemap line to robots.txt.
Do I need robots.txt?
Not strictly, but it is good practice.
More free tools
Meta Tag and On-Page SEO Checker | Schema Markup Generator | All free tools