Check if AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can access your website. Free robots.txt analysis for 14+ AI crawlers — no account needed.
Free · No account needed · 3 checks per day
AI models need to crawl your website to include your content in their knowledge base. If your robots.txt blocks AI crawlers, your content won't appear in ChatGPT, Claude, or Perplexity — regardless of your traditional SEO rankings.
A quick AI crawler check can reveal whether you're accidentally blocking the bots that drive AI search traffic. Most sites block at least 3-5 major AI crawlers without realizing it.
| Crawler | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | ChatGPT knowledge base |
| ClaudeBot | Anthropic | Claude AI |
| PerplexityBot | Perplexity | AI search |
| Google-Extended | GeminiandAI Overviews | |
| OAI-SearchBot | OpenAI | ChatGPT Search |
| CCBot | Common Crawl | Web archive |
| FacebookBot | Meta | AI training |
| Applebot-Extended | Apple | Apple Intelligence |
| Anthropic-AI | Anthropic | Claude training |
| cohere-ai | Cohere | Enterprise AI |
AI crawlers are bots that scan websites to collect data for training or powering AI models like ChatGPT, Claude, and Perplexity. If your robots.txt blocks them, your content won't appear in AI responses.
Enter your URL in the checker above. The tool will analyze your robots.txt and show whether GPTBot, ClaudeBot, and 12+ other AI crawlers are allowed or blocked.
If you want your content to appear in ChatGPT, Claude, and Perplexity responses, you should allow AI crawlers. Blocking them reduces your AI visibility to zero.
GPTBot is used for training OpenAI's models, while OAI-SearchBot powers ChatGPT's search feature. You can allow one and block the other in your robots.txt.
There are 14+ known AI crawlers. The most important ones are GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and OAI-SearchBot. Our free checker covers all of them.
