FREE AI SEARCH CRAWLER & SSR READABILITY TEST$0 · NO ACCOUNT REQUIRED
Free GPTBot, OAI-SearchBot & PerplexityBot WAF Checker
Many sites allow AI crawlers in /robots.txt while unknowingly returning HTTP 403 WAF challenges or empty client-side JavaScript shells to GPTBot and PerplexityBot. Test all 5 AI crawlers and non-JS HTML readability below.
DIAGNOSTIC COVERAGE
What this live HTTP inspector checks
CHECK 01
/robots.txt allow/disallow parsing across GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended
CHECK 02
Live simulated GPTBot/1.2 and PerplexityBot/1.0 User-Agent HTTP probes to catch silent CDN/Cloudflare 403 blocks
CHECK 03
SSR vs. Client-Side JS hydration readability (raw HTML bytes, visible word count, and token signal ratio)
CHECK 04
Auto-detected CMS platform guidance to unblock AI search crawlers safely
PROTOCOL QUESTIONS
Frequently asked questions
Why does Perplexity or ChatGPT skip my site even when /robots.txt says Allow: /?
Two culprits cause most invisible drops: CDN Bot Fight Mode returning HTTP 403 challenges to AI user-agents, or client-side React/SPA rendering that serves an empty <div id="root"> shell to non-JS AI crawlers.
Does this check consume paid API credits?
No. The AI Bot & WAF Checker runs direct HTTP and HTML readiness probes free of charge (up to 15 custom domain checks per IP per day).