Verify robots.txt isn't blocking AI bots. Check CDN settings. Confirm in server logs that AI crawlers reach your content.
There are two types of AI bots: retrieval bots (ChatGPT-User, PerplexityBot) that cite your content to users, and training bots (GPTBot, Google-Extended) that use your content for model training. Allow retrieval bots — blocking them means blocking AI citations of your brand.
Cloudflare's "Bot Fight Mode" and "Super Bot Fight Mode" may block AI crawlers without your knowledge. Check Cloudflare > Security > Bots and add exceptions for AI retrieval bots you want to allow.
Visit yourdomain.com/robots.txt and check for any User-agent rules that might block AI bots. Common AI bot names: ChatGPT-User, GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Amazonbot, anthropic-ai.
Retrieval bots (ALLOW these — they cite you): ChatGPT-User (used when ChatGPT browses the web), PerplexityBot, Googlebot (also powers AI Overviews). Training bots (OPTIONAL to block): GPTBot (OpenAI training), Google-Extended (Gemini training), CCBot.
If your robots.txt has blanket "Disallow: /" rules for unknown bots, add explicit Allow rules for retrieval bots. Example: "User-agent: ChatGPT-User\nAllow: /" and "User-agent: PerplexityBot\nAllow: /".
If you prefer your content not be used for AI model training, add: "User-agent: GPTBot\nDisallow: /" and "User-agent: Google-Extended\nDisallow: /". This is a business decision with no SEO impact.
Log into Cloudflare > Security > Bots. Check if "Bot Fight Mode" is enabled. If so, AI bots may be getting blocked at the CDN level even though your robots.txt allows them. Add firewall rules to allow specific AI bot user agents.
Check your server access logs for AI bot user agents. Search for "ChatGPT-User", "PerplexityBot", "ClaudeBot" in your logs. If they're returning 403 or are absent entirely, something is blocking them (firewall, WAF, rate limiting).
Review my robots.txt for AI crawler access: [PASTE YOUR ROBOTS.TXT] Check if these AI bots are allowed or blocked: 1. ChatGPT-User (OpenAI's retrieval bot) 2. GPTBot (OpenAI's training bot) 3. PerplexityBot 4. ClaudeBot (Anthropic) 5. Google-Extended (Google AI training) 6. Googlebot (regular search + AI Overviews) 7. Bingbot (regular search + Copilot) For each, state if it's ALLOWED or BLOCKED. Recommend which to allow (retrieval) vs. optionally block (training). Generate an updated robots.txt with proper AI bot handling.
Track your progress and get guided through every step.
Open Interactive Tool