AI Crawler Access Checker
When someone asks ChatGPT or Perplexity about your topic, a crawler fetches your pages to build the answer. Three things quietly stop it: robots.txt disallowing a bot, a firewall challenge a bot can't solve, or a server too slow before it gives up. This checks a domain the way a crawler sees it and shows you what's getting through. The background here.
New to this?
If this tool saves you time, you can sponsor the work on GitHub.
How does this work?
Every result is a real request. It sends each crawler's user agent to your site, reads robots.txt, fingerprints your firewall, times your server, and runs a burst test. A spoofed Googlebot acts as a control: if that's challenged too, your firewall is checking bot identity by IP, not blocking, so the real crawlers pass.
What no external tool can see is how your firewall treats the crawlers' actual IP ranges, since it isn't fetching from OpenAI's or Anthropic's network. When the answer depends on that, the tool says so instead of guessing, and points you to your server logs, which are the one place that knows for sure.
Passed? Check these next
- 499 Timeout Risk Checker
Time your pages against the live fetchers’ patience.
- Agentic Resource Discovery Checker
See whether AI agents can discover what your site offers.
- Knowledge Panel Readiness Checker
See whether Google holds an entity for you.