The scraper had been green for eleven weeks. Every run exited 0, every run wrote rows to the warehouse, the dashboard showed a healthy line. Then someone in the pricing team asked why 40% of products suddenly had no price. They'd been shipping decisions on that data for most of a week.
Nothing had failed. That's the whole problem. The site had quietly moved the price into a different element during an A/B test, our selector returned null, and the pipeline did exactly what we told it to: it wrote null and moved on. A scrape can succeed structurally and fail semantically, and almost nobody monitors for the second kind.
Exit 0 is not the same as correct
I've written about scrapers that return zero results and still exit clean. Silent drift is the same disease at a different stage. The request went through. The HTML parsed. The loop ran. Every layer reported success because every layer only knows about its own job, and none of them know what a good record is supposed to look like.
Markup drift is the number one way scrapers rot, and it rarely announces itself with a crash. A crash would be a gift. A crash pages someone. What actually happens is subtler: a retailer ships a redesign, or rolls out a variant to 10% of traffic, or renames a CSS class in a build step, and one field goes dark while the other nine keep flowing. Your fill rate drops from 99% to 61% and the only thing that changed color is a number in a spreadsheet three teams away.
