Crawlerdesk
Sample
Dated internal sample. Not a paid delivery.
Public /robots.txt only. Named crawlers: GPTBot and Google-Extended in this cut. Not legal advice. Not a claim that any crawler honors the file.
| Host | robots.txt | GPTBot named | Google-Extended named |
|---|---|---|---|
| www.nytimes.com | 200 | yes | yes |
| www.washingtonpost.com | 200 | no | no |
| www.theguardian.com | 200 | no | no |
| www.cloudflare.com | 200 | yes | yes |
| www.openai.com | 200 | no | no |
| www.anthropic.com | 200 | no | no |
Internal sample used to validate the page. Checkout is not live. Price hypothesis $49.