Short answer: a sitemap lists the pages you want search engines to find. robots.txt tells crawlers which parts of your site they may not visit. One invites, the other keeps out, and the best setup uses both: robots.txt also tells search engines where your sitemap is.
| Sitemap | robots.txt | |
|---|---|---|
| Purpose | List the pages that should be found | Keep crawlers out of parts of the site |
| Format | XML file, any name, usually sitemap.xml | Plain text, always /robots.txt in the root |
| Required? | No, but it helps search engines find every page | No; without one, crawlers may visit everything |
| Keeps a page out of Google? | No | No: it stops crawling, not indexing |
How they work together
Add one line with the full address of your sitemap to robots.txt:
Sitemap: https://www.example.com/sitemap.xml
Every search engine that reads your robots.txt now knows where your sitemap is, without you submitting it anywhere. The line can go anywhere in the file.
Mistakes to avoid
- Listing blocked pages in your sitemap. A page that robots.txt blocks cannot be crawled, so it has no business in your sitemap. Search Console reports it as "Submitted URL blocked by robots.txt".
- Using robots.txt to hide a page. A blocked page can still show up in search results, without a
description, when other sites link to it. To keep a page out, let it be crawled and give it a
noindex tag:
<meta name="robots" content="noindex">. - Listing noindex pages in your sitemap. The sitemap says "index this", the page says "don't". Leave noindex pages out.
- Blocking your sitemap. robots.txt must not block the sitemap file itself.
And our spider?
Free Sitemap Generator reads your robots.txt before it crawls and leaves blocked pages out of your sitemap, just like a search engine would. The sitemap validator checks an existing sitemap for addresses that your robots.txt blocks.