Three ETL failure patterns I now write into the output file, not just the logs

My Reddit scraper returned empty arrays for 71 days. Nothing broke visibly. The market-listening output file kept being written, kept being committed to git, kept looking like a healthy daily snapshot. Nobody caught it until the interpretation layer noticed that Reddit-sourced signals hadn't changed in two months.

The three patterns I added afterward are small. Combined, they mean that failure mode can't happen silently again — the artifact itself reports what went wrong, not just the runner logs that nobody reads unless something is already on fire. All three are implemented in scripts/market-listening/collect.mjs, with the 71-day blind spot documented in the inline comments.

Pattern 1: sources_ok lives in the artifact, not the logs

The daily output JSON now contains a sources_ok object with a boolean per source:

{

Three ETL failure patterns I now write into the output file, not just the logs

Related reading

I Ship a New Data Scraper Every Few Days. Here Is What I Have Learned

Your scraper isn't broken. The site changed, and it didn't tell you.

"[Jul 27-29] A Silent Crash, a Missing Record, and 27 Hours of Signal Silence"

Three checks that lied to me while building a content extraction API

Your Scraper Collected 50 Rows. There Were 4,000.

Schema Drift Is the Silent Killer. Here's What to Log So You Actually Catch It.