Your crawl job finished successfully. That doesn't mean it got the data.
Every scraping pipeline has a monitoring dashboard, and every monitoring dashboard has the same blind spot: it tracks whether requests succeeded, not whether the content that came back was real. A job that completes with a wall of green 200 status codes looks healthy. It can also be quietly wrong, page after page, for weeks, because a 200 response only tells you the server accepted the request. It says nothing about whether you're looking at the actual page or a version built specifically for visitors the site doesn't fully trust.
That gap between "the request succeeded" and "the data is correct" is where most silent pipeline failures live, and it's getting wider as anti-bot systems get more sophisticated about what they serve instead of an outright block.
What a "successful" response can actually contain
A block used to be simple to detect: a 403, a 429, a connection reset. Modern anti-bot systems increasingly prefer a different approach, because an obvious block tells the requester exactly what happened and invites a fix. A soft block, served with a 200, doesn't.
