Besides search engines, a new kind of visitor reads your website these days: crawlers that collect text for AI models and AI assistants. Like search engine spiders, they announce themselves with a name, and like search engine spiders, the well-behaved ones obey your robots.txt.
The names you will see most:
- GPTBot, from OpenAI.
- ClaudeBot, from Anthropic.
- CCBot, from Common Crawl, whose public archive many AI models learn from.
- PerplexityBot, from Perplexity.
- Google-Extended, which is not a separate crawler but a name you can use in robots.txt to tell Google not to use your pages for its Gemini models. It does not change how your site appears in Google Search.
To keep one of them out, give it its own group in robots.txt:
User-agent: GPTBot Disallow: /
Whether you allow them is up to you: blocking them keeps your text out of future models, allowing them can bring your site into the answers people get from AI assistants. Your sitemap does not change any of this; a crawler that may not visit your site will not read your sitemap either.
You may also come across llms.txt, a proposal from late 2024 for a short text file at the root of your site that tells AI assistants what the site is about and where its important pages are. It is not a standard yet, and adding one is optional.