Free tool · no sign-up
llms.txt checker and AI crawler robots.txt test
Paste your site. You see whether your llms.txt is real and complete, which AI crawlers your robots.txt lets in, and the exact lines to paste where something is off.
What it checks
- llms.txt. We fetch
/llms.txtand check it is a real text file, not your home page answering for it, then look for the shape llmstxt.org proposes: a# Nametitle, a>summary line and links to your key pages. We also note whether/llms-full.txtexists. - robots.txt. We read your rules the way the robots.txt standard (RFC 9309) says a crawler must: a crawler follows the groups that name it and ignores the
*group, and the longest matching rule wins. Then we report each AI crawler as allowed, partly blocked, limited to a few pages or the home page, or blocked, with any Crawl-delay, and whether a Sitemap line points to your sitemap. - Fixes. Copy-paste robots.txt lines and a starter llms.txt built from your home page title and description.
The search and answer crawlers are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot. The training crawlers are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider. If your site sits behind bot protection that answers us with a challenge, we say we could not read the file. We never count that as allowed.
More on the crawlers and a full example file: robots.txt for AI crawlers. For agents: GET https://snoffle.broke2builtai.com/api/llms-check?url=example.com returns the same answer as JSON (OpenAPI).
Quick answers
- What is llms.txt?
- A short Markdown file at yoursite.com/llms.txt that tells AI tools what the site is and links its most useful pages. It is a proposed standard from llmstxt.org. Not every AI tool reads it yet, so it is cheap insurance, not a switch that gets you recommended.
- How do I let ChatGPT, Claude and Perplexity read my site?
- Serve a real robots.txt at the root of your site that does not block their crawlers: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot and Perplexity-User for Perplexity. A firewall that challenges every visitor can block them too, even when robots.txt allows them.
- Should I block GPTBot and other training crawlers?
- That is your call. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and similar crawlers collect training data. Blocking them does not stop the search and answer crawlers, which are separate names.
- Does passing this check mean AI assistants will recommend me?
- No. It only shows that the two files let AI crawlers in and describe your site. Whether an assistant names you depends on what it finds when a buyer asks, which is what the full Snoffle report measures.