Originally published at crawlora.net.
When you hit a dead URL in production, do you know whether the domain is gone — or whether an anti-bot system just blocked your crawler? They look identical from a failed request, but they're completely different failures, and most tools don't tell them apart.
We scanned the DomCop top 10 million domains to find out how much of the popular web is actually dead. The short version: about 14% — not the ~27% you've probably seen quoted.
Dead and blocked are not the same failure
A domain that won't load failed for one of two reasons:
