The mess
robots.txt is a file, not a dashboard.
GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot — named or not, allow or disallow — is buried in a text file most teams never re-read.
For publishers and SEO shops
Validating
You send a public URL. We fetch robots.txt only and return a dated matrix of named AI crawlers mentioned or blanket-disallowed. Observational. Not a scrape of the rest of the origin.
Price hypothesis $49. Checkout is not live. We are not claiming a proven business.
The mess
GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot — named or not, allow or disallow — is buried in a text file most teams never re-read.
The file
Host, HTTP status, which named crawlers appear, and whether a blanket disallow is present. One URL in. One report out.
The limit
Fetching robots.txt is the protocol access method. It is not permission to scrape the site, not a training opt-out, and not a guarantee any crawler honors the file.