free tool · 03
AI Crawler Access Checker
This checker reads your robots.txt and llms.txt, tells you which of the 10 AI crawlers that matter — from OpenAI's GPTBot to Anthropic's ClaudeBot and Perplexity's PerplexityBot — can actually access your site, and then live-fetches your page under the retrieval bots' user agents to catch the blocks robots.txt can't see: CDN and bot-protection rules that 403 a crawler your robots.txt allows. A crawler that can't reach you is an engine that can never cite you. Results are instant and free.
The 10 crawlers we check
| User agent | Who it is |
|---|---|
| GPTBot | OpenAI — model training |
| OAI-SearchBot | OpenAI — ChatGPT search index |
| ChatGPT-User | OpenAI — live browsing on user request |
| ClaudeBot | Anthropic — crawling |
| Claude-SearchBot | Anthropic — search index |
| Claude-User | Anthropic — live fetches for users |
| Google-Extended | Google — Gemini training/grounding |
| GrokBot | xAI — Grok (no official docs; reported by third-party trackers) |
| PerplexityBot | Perplexity — answer engine |
| CCBot | Common Crawl — used in many training sets |
One caveat: xAI publishes no official crawler documentation, so the GrokBot entry comes from third-party bot trackers rather than from xAI. We still check it because many robots.txt files reference it explicitly. The other nine bots are documented by their operators.
One wildcard rule can hide you from every engine
A single Disallow: / under User-agent: * — often left over from a staging config or a bot-blocking plugin — silently removes you from AI answers across all four major assistants. The fix usually takes two minutes.
Access is step one. Visibility is the goal.
See whether ChatGPT, Gemini, Claude, and Grok actually mention your brand — free.
Check my visibility