Crawlerdesk

Product

Public URL in. Dated crawler matrix out.

Intended customer: small publishers and SEO agencies who need a readable snapshot of AI crawler rules on a site they own or advise.

What the report contains

  1. The host you named.
  2. Whether /robots.txt returned HTTP 200.
  3. Which of a fixed named-crawler list appear in that file.
  4. Whether a blanket Disallow: / is present for those agents.
  5. The fetch date. The raw robots.txt is not republished.

What it is not

Not a crawl of HTML. Not /llms.txt. Not a pentest. Not a ranking product. Not legal advice. Not a claim that crawlers obey robots.txt.

See the sample · Pricing hypothesis