Generators · 100% free

Ecommerce robots.txt Generator

Generate a robots.txt with ecommerce-aware defaults, block faceted filters, allow checkout assets, manage AI crawlers.

Shopify WooCommerce BigCommerce Any ecommerce

Sitemap URL

AI crawler controls

GPTBot (OpenAI)

ChatGPT training crawler

OAI-SearchBot (OpenAI)

ChatGPT search citations

ChatGPT-User (OpenAI)

Live browse / agent traffic

ClaudeBot (Anthropic)

Claude training + search

PerplexityBot

Perplexity AI search

Google-Extended (Gemini)

Bard/Gemini training

Bytespider (TikTok)

Doubao / TikTok AI

CCBot (Common Crawl)

Bulk web archive used by many AIs

Applebot-Extended

Apple Intelligence training

Extra disallow paths

Extra allow paths

text

# Generated by aishoppingfeeds.com/tools/ecommerce-robots-txt-generator

User-agent: *
Disallow: /checkout
Disallow: /checkouts/
Disallow: /cart
Disallow: /account
Disallow: /account/
# Faceted filter URLs (color, size, price etc.)
Disallow: /*?filter*
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?price=
Disallow: /*?sort_by=

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://yourstore.com/sitemap.xml

Place this at the root of your domain (e.g. yourstore.com/robots.txt). Test in Google's robots.txt tester before deploying — bad rules can de-index your store overnight.

About this tool

A bad robots.txt can de-index half your store overnight. This generator gives you ecom-aware defaults: block infinite faceted URLs (?color=, ?size=), keep important paths open, and decide which AI crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended) you want to allow or block.

Why ecommerce robots.txt is hard

Ecommerce platforms generate near-infinite URLs from faceted filters (?color=red, ?size=medium, ?sort_by=price-asc). Letting Google crawl all of them wastes crawl budget on duplicate content and hides important product pages. The right defaults block facets, search and account paths while keeping product, collection and content pages discoverable.

The AI crawler decision

Whether to allow GPTBot, ClaudeBot, Perplexity etc. is now a strategic brand decision. Allowing them means your products get cited in AI shopping answers (becoming more important as AI commerce grows). Blocking them protects content from training but invisible-ifies you in agentic shopping. Most ecommerce brands should allow citation crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot for search) even if they block pure training bots.