VERISOURCEmcp.verisource.dev/mcp
← /benchmarksEMPIRICAL HTTP RESEARCH · NOVEMBER 2026N = 40 PUBLIC DOMAINS · 4 VERTICALS

2026 AI Search & Agent Readiness Benchmark: /llms.txt, /agents.md & Silent GPTBot WAF Blocks.

EXECUTIVE EMPIRICAL SUMMARY · VERIFIED HTTP TELEMETRY (2026-11-16)

Across 40 live-probed B2B SaaS, Shopify D2C, SEO agency, and multi-location service domains in November 2026, 82.5% ship homepage JSON-LD, 50.0% publish a custom /llms.txt (while another 17.5% serve unedited platform boilerplate and 7.5% return HTML soft-404s), 7.5% include homepage FAQPage schema, 5.0% publish a custom /agents.md, and 5.0% silently block GPTBot via HTTP 403 WAF rules despite an open /robots.txt.

Custom /llms.txt
50.0%
20/40 curated · 7/40 boilerplate · 3/40 soft-404
Custom /agents.md
5.0%
2/40 custom (vercel.com, linear.app) · 5/40 stock Shopify
Homepage FAQPage Schema
7.5%
3/40 include FAQPage vs. 33/40 (82.5%) with base JSON-LD
Silent Edge WAF 403 Blocks
5.0%
2/40 return HTTP 403 to GPTBot despite 0/40 blocking in /robots.txt
Cross-Vertical Readiness Matrix (N = 40 Live-Probed Public Domains)Probed 2026-11-16 · 6–8 live HTTP requests
Industry CohortSampleCustom /llms.txtAuto BoilerplateCustom /agents.mdBase JSON-LDFAQPage SchemaWAF HTTP 403
B2B SaaS & Developer Tools1010/10 (100%)0/10 (0%)2/10 (20%)9/10 (90%)0/10 (0%)0/10 (0%)
Shopify & D2C Commerce101/10 (10%)6/10 (60%)0/10 (0%)7/10 (70%)0/10 (0%)1/10 (10%)
SEO, Content & Digital Agencies104/10 (40%)1/10 (10%)0/10 (0%)8/10 (80%)2/10 (20%)0/10 (0%)
Multi-Location Healthcare, Legal & Home Services105/10 (50%)0/10 (0%)0/10 (0%)9/10 (90%)1/10 (10%)1/10 (10%)
All Cohorts Combined4020/40 (50.0%)7/40 (17.5%)2/40 (5.0%)33/40 (82.5%)3/40 (7.5%)2/40 (5.0%)
KEY ARCHITECTURAL TAKEAWAYS

Four protocol gaps separating cited brands from invisible domains

80% UCP ACTIVE VS. 0% CUSTOM /AGENTS.MD

The 80-Point Shopify Agent Truth Gap

Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.

0% ROBOTS.TXT BLOCK · 5.0% WAF HTTP 403

Open /robots.txt Files Mask Silent CDN 403 Blocks

Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real GPTBot or PerplexityBot User-Agent headers at the CDN edge.

50.0% CURATED · 17.5% BOILERPLATE · 7.5% SOFT-404

One-Third of HTTP 200 /llms.txt Responses Are Uncurated or HTML Soft-404s

Thirty domains returned HTTP 200 on GET /llms.txt, but Content-Type and body inspection revealed 3 HTML SPA soft-404 pages (7.5% of the cohort) and 7 auto-generated Shopify or Rank Math boilerplate dumps (17.5%). Developer SaaS leads genuine curation at 100% (10/10), compared with 40% of SEO agencies and 10% of D2C brands.

82.5% BASE JSON-LD · ONLY 7.5% FAQPAGE

Base Organization Schema Is Common; Answer-First FAQPage Schema Is Rare

While 33 of 40 domains (82.5%) render at least one application/ld+json block on their homepage, only 3 of 40 (7.5%) include FAQPage structured Q&A pairs—leaving ChatGPT Search, Gemini, and Perplexity to infer pricing tiers, service areas, and SLAs from third-party directories.

DOMAIN-LEVEL HTTP DATASET · REPRODUCIBLE TELEMETRY
Filter all 40 probed domains by industry vertical
SHOWING 40 OF 40 DOMAINS
DomainVertical/llms.txt/agents.mdHomepage JSON-LDGPTBot / PerplexityBotShopify UCPLive Check
stripe.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
vercel.comB2B SaaSCUSTOM CURATEDCUSTOM ENRICHED3 nodesHTTP 200 OPEN—Re-Probe →
supabase.comB2B SaaSCUSTOM CURATEDMISSING4 nodesHTTP 200 OPEN—Re-Probe →
resend.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
linear.appB2B SaaSCUSTOM CURATEDCUSTOM ENRICHED0 nodesHTTP 200 OPEN—Re-Probe →
cloudflare.comB2B SaaSCUSTOM CURATEDMISSING3 nodesHTTP 200 OPEN—Re-Probe →
posthog.comB2B SaaSCUSTOM CURATEDMISSING2 nodesHTTP 200 OPEN—Re-Probe →
clerk.comB2B SaaSCUSTOM CURATEDMISSING2 nodesHTTP 200 OPEN—Re-Probe →
dub.coB2B SaaSCUSTOM CURATEDMISSING2 nodesHTTP 200 OPEN—Re-Probe →
cal.comB2B SaaSCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
allbirds.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY0 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
gymshark.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY0 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
brooklinen.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY4 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
chubbies.comShopify D2CCUSTOM CURATEDMISSING1 nodeHTTP 200 OPENUCP ACTIVERe-Probe →
ridge.comShopify D2CAUTO BOILERPLATEMISSING0 nodesBOTH WAF 403UNAVAILABLERe-Probe →
cutsclothing.comShopify D2CMISSINGMISSING2 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
ruggable.comShopify D2CHTML SOFT-404MISSING1 nodeHTTP 200 OPENUNAVAILABLERe-Probe →
trueclassictees.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY3 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
carawayhome.comShopify D2CMISSINGMISSING2 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
beardbrand.comShopify D2CAUTO BOILERPLATESTOCK SHOPIFY2 nodesHTTP 200 OPENUCP ACTIVERe-Probe →
siegemedia.comSEO AgencyMISSINGMISSING2 nodesHTTP 200 OPEN—Re-Probe →
animalz.coSEO AgencyMISSINGMISSING0 nodesHTTP 200 OPEN—Re-Probe →
omniscientdigital.comSEO AgencyMISSINGMISSING1 nodeHTTP 200 OPEN—Re-Probe →
singlegrain.comSEO AgencyCUSTOM CURATEDMISSING3 nodes · FAQPageHTTP 200 OPEN—Re-Probe →
directiveconsulting.comSEO AgencyCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
victorious.comSEO AgencyMISSINGMISSING2 nodes · FAQPageHTTP 200 OPEN—Re-Probe →
webmechanix.comSEO AgencyAUTO BOILERPLATEMISSING5 nodesHTTP 200 OPEN—Re-Probe →
klientboost.comSEO AgencyHTML SOFT-404MISSING0 nodesHTTP 200 OPEN—Re-Probe →
powerdigitalmarketing.comSEO AgencyCUSTOM CURATEDMISSING2 nodesHTTP 200 OPEN—Re-Probe →
webfx.comSEO AgencyCUSTOM CURATEDMISSING3 nodesHTTP 200 OPEN—Re-Probe →
aspendental.comMulti-LocationHTML SOFT-404MISSING2 nodesHTTP 200 OPEN—Re-Probe →
hellotend.comMulti-LocationMISSINGMISSING36 nodesHTTP 200 OPEN—Re-Probe →
asktia.comMulti-LocationMISSINGMISSING1 node · FAQPageHTTP 200 OPEN—Re-Probe →
onemedical.comMulti-LocationMISSINGMISSING0 nodesHTTP 200 OPEN—Re-Probe →
forthepeople.comMulti-LocationCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
rotorooter.comMulti-LocationCUSTOM CURATEDMISSING3 nodesHTTP 200 OPEN—Re-Probe →
servpro.comMulti-LocationCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
mrrooter.comMulti-LocationCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
aireserv.comMulti-LocationCUSTOM CURATEDMISSING1 nodeHTTP 200 OPEN—Re-Probe →
westerndental.comMulti-LocationMISSINGMISSING2 nodesGPTBOT WAF 403—Re-Probe →
METHODOLOGY & REPRODUCIBILITY SPECIFICATION

How every domain was probed and classified

Each of the 40 public domains was probed directly over HTTPS using the same deterministic parser powering Verisource's production scanner: (1) GET /robots.txt parsed for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended directives; (2) homepage requests sent with explicit GPTBot/1.2 and PerplexityBot/1.0 User-Agent headers to catch silent CDN HTTP 403 / 429 blocks; (3) GET /llms.txt and GET /agents.md inspected for text/html SPA soft-404s and stock Shopify (shop.app/SKILL.md) or SEO plugin boilerplate; (4) homepage application/ld+json script nodes parsed for FAQPage entities; and (5) JSON-RPC tools/list handshakes tested against /api/ucp/mcp and /api/mcp on D2C commerce stores.

Free /llms.txt & /agents.md Validator →Free GPTBot & PerplexityBot WAF Checker →Free Shopify /api/ucp/mcp Checker →Copy 7-CMS Starter Templates
NOVEMBER 2026 ADDENDUM · DEVELOPER INFRASTRUCTURE COHORT (N = 10)

90% of DevTools ship /llms.txt. Only 20% ship /agents.md.

In a follow-up live HTTP probe across 10 developer infrastructure and B2B SaaS domains (trigger.dev, inngest.com, turso.tech, tinybird.co, knock.app, workos.com, stytch.com, upstash.com, highlight.io, and infisical.com), 9 of 10 (90.0%) served a valid /llms.txt directory, yet only 2 of 10 (20.0% — tinybird.co and infisical.com) published a custom /agents.md pricing and limits guardrail file, and only 1 of 10 (10.0% — upstash.com) shipped all 3 JSON-LD pillars.

Protocol / Schema SignalDevTools Cohort (N = 10)Commercial Impact
/llms.txt (HTTP 200)90.0% (9 / 10)High documentation-link adoption across modern DevTools
Custom /agents.md20.0% (2 / 10)80% leave pricing tiers & rate limits unguarded against AI hallucination
Full 3-Pillar JSON-LD10.0% (1 / 10)90% omit FAQPage or Offer pricing nodes on root pages
BENCHMARK FAQ

Common questions about the 2026 AI Search Readiness dataset

What percentage of websites have /llms.txt and /agents.md in November 2026?

In our live HTTP readiness probe across 40 public domains spanning B2B SaaS, Shopify D2C, SEO agencies, and multi-location services, 50.0% (20/40) serve a custom-curated /llms.txt file, another 17.5% (7/40) expose auto-generated Shopify or SEO-plugin boilerplate, 7.5% (3/40) return HTML soft-404s with HTTP 200 status, and only 5.0% (2/40 — vercel.com and linear.app) publish a custom /agents.md specification.

Why do Shopify stores show 80% /api/ucp/mcp adoption but 0% custom /agents.md adoption?

Shopify rolled out platform-level /api/ucp/mcp endpoints and stock /agents.md files pointing to https://shop.app/SKILL.md across modern storefronts, which explains why 8 of 10 probed D2C stores respond on /api/ucp/mcp. However, stock Shopify files omit brand-specific free-shipping thresholds, return windows, and bundle exclusions unless the merchant overrides them via templates/agents.md.liquid or /pages/ai-facts.

Can a domain block GPTBot or PerplexityBot even when /robots.txt allows them?

Yes. Zero of the 40 probed domains disallowed GPTBot or PerplexityBot at root in /robots.txt, yet 5.0% (2/40 — ridge.com and westerndental.com) returned HTTP 403 Forbidden at the CDN/WAF layer when requested with GPTBot or PerplexityBot User-Agent headers. Testing both /robots.txt and simulated crawler User-Agent HTTP status codes is required.

BENCHMARK YOUR OWN DOMAIN IN 60 SECONDS

See how your website stacks up against the 40-domain 2026 readiness cohort.

$0 TODAY · 7-DAY FREE TRIAL
OR TEST A LIVE BENCHMARK DOMAIN:stripe.comlinear.appcal.comshopify.com