A scheduled job has two failure modes. The loud one throws an error and your alerting catches it. The quiet one just stops running, and nothing catches it, because your alerting is wired to the job itself. No job, no error, no alert. The data it maintains gets staler by the day and every downstream system keeps serving it like nothing's wrong.
I run a handful of production systems by myself. One of them is a niche content site with a curated data feed behind it, plus a release-tracking pipeline on Cloudflare Workers and D1, plus some geolocation alerting. There's no ops team. The quiet failure mode is the one that actually bit me.
How I found out
The curation job that refreshes the feed died without a sound. No exception. Nothing in the logs that would make you stop scrolling. The site kept serving the last good copy of the feed, so every health check I had stayed green — the checks were all downstream of the dead job, and downstream was fine.
I found out days later, reading my own site as a user, thinking huh, this hasn't changed in a while.
