Byte Size Summary
Most pipeline failures are transient — a registry returning a 503, a smoke test catching a slow cold start, a network blip during an image push. Retrying them automatically, with exponential backoff, means engineers never see them. The failures that reach a human should be the ones that actually need one. This article builds a retry wrapper and a three-tier alerting system (transient → silent, degraded → Slack warning, critical → PagerDuty page) on top of a GitHub Actions blue/green deploy workflow. The demo application is Waybill — a FastAPI shipment tracking API backed by PostgreSQL, where the health endpoint checks real database connectivity rather than returning a static 200. That distinction matters: a smoke test that only checks HTTP status is a smoke test that passes while your database is unreachable. By the end you will have a working repo you can run locally with Docker Compose and test today.
The Story
There is a specific kind of 11pm message that every engineer eventually receives.
Pipeline failed.
