How do I allow GPTBot and ClaudeBot in robots.txt?

Add "User-agent: GPTBot" and "User-agent: ClaudeBot" lines, each followed by "Allow: /", to a real robots.txt file served at the root of your site, and point it at your sitemap.

Why does my robots.txt show my homepage?

Your app is answering every unknown address with the homepage, so the file doesn't exist. Put robots.txt in your public folder so it is served as a plain text file.

Guide · updated 2026-10-31

robots.txt for AI crawlers: GPTBot, ClaudeBot, PerplexityBot

robots.txt is a small text file at the root of your site that tells crawlers what they may read. If you want AI tools to recommend you, they need to be allowed in, and the file has to really exist.

Check yours first

Open yoursite.com/robots.txt. You should see a few lines of plain text. If you see your homepage instead, the file doesn't exist and your app is answering for it; many Lovable, Bolt and React apps do this. Put the file in public/robots.txt so it's served as-is.

A robots.txt that welcomes AI tools

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

This is the same pattern Snoffle serves for itself (see it). Change the sitemap line to your own address.

What these names are

GPTBot, OAI-SearchBot, ChatGPT-User: OpenAI's crawlers for training, for search results in ChatGPT, and for fetching a page when a user asks.
ClaudeBot, Claude-SearchBot, Claude-User: Anthropic's equivalents.
PerplexityBot: Perplexity's search crawler.
Google-Extended: controls whether Google uses your content for its AI models; normal Google Search uses Googlebot either way.

Allowing a crawler doesn't make anyone recommend you. Blocking one, or having no real file, can quietly keep you out.

Quick answers

How do I allow GPTBot and ClaudeBot in robots.txt?: Add "User-agent: GPTBot" and "User-agent: ClaudeBot" lines, each followed by "Allow: /", to a real robots.txt file served at the root of your site, and point it at your sitemap.
Why does my robots.txt show my homepage?: Your app is answering every unknown address with the homepage, so the file doesn't exist. Put robots.txt in your public folder so it is served as a plain text file.

See exactly what's stopping your site

See a real report first · Teams: Pro $499 a month, Agency $1,500 a month.