As mentioned on november 22nd, we wanted to include support for robots.txt files. So now robots.txt files are fetched and obeyed! It was prety useless to spider parts of your site to put in a sitemap if other spiders (Googlebot et al) are not able to fetch does pages too. It's still in beta and tested further. Feedback is appreciated.