Our spider finds your pages the way a search engine does: it starts at your homepage and follows the links it finds. A page that is missing from your sitemap is almost always a page the spider could not reach, or was not allowed to. The usual reasons:
- No link points to it. A page that is only reachable through a search box, a form or a menu built with JavaScript cannot be found by following links. Link to it from a normal page.
- The link goes somewhere else. We only follow links that start with the address you gave us. If you entered https://www.example.com/, links to http://example.com/ are a different site.
- robots.txt says no. We obey your robots.txt, so pages you block there stay out of your sitemap too.
- The page gave an error. Only pages that load correctly go into your sitemap. Pages that return "not found" or a server error are listed in your control panel instead.
- The limit was reached. Every sitemap job has a maximum number of pages. Premium accounts can go further.
Tip: the error list in your control panel often shows the answer, and the quick sitemap tool lets you check a single page and the links on it.