AI Crawler robots.txt Audit
Paste your robots.txt to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) you're allowing or blocking.
About this tool
Whether you allow or block AI crawlers is now a strategic decision. This audit parses your existing robots.txt and reports each major AI bot's access, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended (Gemini), Bytespider, CCBot, Applebot-Extended, so you can choose your posture deliberately.
Training bot vs citation bot — they're different
OpenAI runs GPTBot (training), OAI-SearchBot (search index) and ChatGPT-User (live browse) as three separate user agents. Blocking GPTBot doesn't block OAI-SearchBot — and citations in ChatGPT come from the search index, not the training data. Same split for Anthropic (ClaudeBot covers both today but expect split), Google (Google-Extended training vs the regular Googlebot), and Apple (Applebot-Extended training).
The pragmatic default for ecommerce
- • Allow citation bots (OAI-SearchBot, PerplexityBot, ChatGPT-User, regular Googlebot)
- • Block or allow training bots based on whether you want your content shaping the next model generation
- • Block aggressive content-scraping bots (Bytespider, CCBot if not needed)
Frequently asked questions
Should I block AI crawlers entirely? +
Depends on your goals. Blocking training bots (GPTBot, ClaudeBot training, Google-Extended) protects content from being used in next-generation models. But blocking citation/search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) makes you invisible in AI shopping answers. Most ecommerce brands should allow citation bots while choosing on training.
What's the difference between GPTBot and ChatGPT-User? +
GPTBot = bulk scraping for training data. ChatGPT-User = live fetches when a user asks ChatGPT to browse a page. OAI-SearchBot = OpenAI's search index crawler. Three separate user agents, three separate decisions.
Will blocking these crawlers hurt my SEO? +
Not for traditional Google search, Googlebot is separate from Google-Extended (Gemini training). Blocking Google-Extended only affects whether your content trains Gemini, not whether you rank in Google. Same for Applebot vs Applebot-Extended.
Can I block training bots but allow search bots? +
Yes, that's the most common posture for ecommerce. Block: GPTBot, Google-Extended, Applebot-Extended, ClaudeBot, Bytespider, CCBot. Allow: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User. Use our Ecommerce robots.txt Generator to build the file with this preset.
More free tools
See all tools →Benchmark your email/SMS popup conversion rate against vertical-specific medians.
Generate a WCAG 2.1 AA accessibility checklist, perceivable, operable, understandable, robust principles.
Find URLs in text that have obvious format errors (no protocol, double slashes, broken patterns).
Compare two blocks of text and highlight what changed, line by line, in-browser.
Why wait? Try it free today.
Stop managing feeds manually. Start optimising with AI in 30 seconds.
- 100% free forever, no credit card required
- 1 brand, 1 feed, 100,000 products per feed
- Full AI Product Optimisation, Rule Engine, and 200+ channel exports
- Pay only for AI credits when you need them