Check if your website has a valid llms.txt file — the standard that helps AI crawlers like GPTBot and ClaudeBot find your most important content.
Free · No account needed · 3 checks per day
Placed at /llms.txt, this file helps AI crawlers understand your site structure and find your most important content. Most sites don't have one — giving you a quick advantage in AI search visibility.
The llms.txt standard was created to solve the problem of AI crawlers wasting resources crawling thousands of low-value pages when they should be focused on your core content. It's like a sitemap, but specifically for AI systems.
# Your Site Name > Brief description of your site for AI context ## Core Pages - Docs: https://yoursite.com/docs - Blog: https://yoursite.com/blog - About: https://yoursite.com/about ## Documentation - Getting Started: https://yoursite.com/docs/getting-started - API Reference: https://yoursite.com/docs/api
A good llms.txt file includes sections for core pages, documentation, and blog content — each with a brief description of what the page contains.
llms.txt is a standard file placed at /llms.txt that helps AI crawlers understand your site structure. It tells GPTBot, ClaudeBot, and other AI crawlers which pages are most important, similar to how robots.txt guides search engine crawlers.
Most sites don't have an llms.txt file, giving early adopters a significant advantage in AI search visibility. It helps AI crawlers find your best content quickly and understand how your site is organized.
Robots.txt tells crawlers what NOT to access. llms.txt tells AI crawlers what TO read and how to prioritize your content. They work together — robots.txt for access control, llms.txt for content guidance.
Place it at the root of your domain: https://yoursite.com/llms.txt. It should be a plain text file following the llms.txt standard format with sections for core pages, documentation, and blog content.
llms.txt doesn't directly improve traditional SEO, but it significantly improves AI visibility. As more users get answers from AI assistants, having your content properly indexed by AI crawlers becomes crucial for traffic.
