A crawlability check often begins with a comforting result: request the homepage, receive HTTP 200, and declare the site open. That conclusion is too broad. The response only proves that one client, from one network, with one user agent and one request shape, reached one representation of the page at one moment.
AI crawlers do not necessarily share any of those conditions. A CDN may classify their user agents differently. A robots rule may apply to one bot but not another. A challenge page may return 200 while replacing the actual document. Even when the HTML is real, the canonical URL or robots metadata can make the page unusable for discovery.
The practical fix is to stop treating the status code as the verdict. Treat it as one item in an evidence chain.
The same URL can represent different access paths
Consider three requests to the same page. A normal browser user agent receives the product page. A generic script receives a managed challenge. A named crawler receives a 403 from a WAF rule. The URL has not changed, but the effective access policy has.
