Crawler SpecificationREAD-ONLY · ROBOTS.TXT OBEYEDGET & HEAD ONLY
VerisourceScan
If this crawler name showed up in your Cloudflare, Vercel, or Nginx logs, someone ran an on-demand site health and AI readiness check on your domain—either from our home-page scanner or via npx @verisource/cli scan on their own machine. This page documents every request the scanner sends and how to block it in 2 lines.
01 · User-Agent Identification
How it identifies itself
Mozilla/5.0 (compatible; VerisourceScan/1.0; +https://verisource.dev/scanner)Every automated HTTP request carries this exact User-Agent header. It never spoofs consumer browsers or third-party search crawlers.
02 · Scope & Crawl Limits
What VerisourceScan checks
- Fetches
/robots.txtfirst and respects bothDisallowpaths andCrawl-delaydirectives. - Inspects structured AI retrieval endpoints (
/llms.txt,/agents.md,JSON-LDentity/FAQ schemas, and Shopify/api/ucp/mcp). - Follows same-origin HTML links up to 5 pages (1 link deep) from the unauthenticated home-page box, or up to 20 pages (2 links deep) inside verified workspaces and CLI runs.
- Verifies linked pages, scripts, stylesheets, images, and
#anchortargets usingHEADrequests where supported, flagging HTTP4xx/5xxresponses and dead links.
03 · Hard Safety Boundaries
What it never does
- Never sends
POST,PUT,PATCH, orDELETEmutations during site health crawls (GETandHEADonly). - Never submits forms, clicks destructive buttons, or transmits session cookies or credentials.
- Never follows paths matching state-changing actions:
logout,signout,delete,remove,unsubscribe, orcancel. - Refuses all loopback, RFC 1918 private networks, cloud metadata IPs (
169.254.169.254), and non-standard ports on every redirect hop.
04 · Opt-Out Instructions
How to keep VerisourceScan out
Add these two lines to your domain’s root /robots.txt file to block all automated VerisourceScan checks immediately:
User-agent: VerisourceScan
Disallow: /