We keep a ledger of every failure from our scheduled agent jobs. It has 72 entries, and 46 of them happened at the git push step — not in the model, not in the API call the job existed to make. Unattended agents mostly fail at writing the result down.
Our store's pipeline runs without a human in the loop for most of the day. Thirteen GitHub Actions workflows are on a cron schedule, carrying eighteen cron entries between them, and each one wakes up a CLI that talks to an external platform and then commits the result back to main. Five of those cron entries belong to a single workflow that posts to Bluesky at five fixed times a day. Two belong to the workflow that publishes an article and announces it.
There is no person watching any of that. That is the point of it — but it also means every assumption in those jobs gets tested at 3 a.m. with nobody around to notice when one of them turns out to be wrong. This is what we have learned from running it that way since June, organized around the ledger, because the ledger disagreed with our intuitions in a useful way.
The ledger says the interesting failures are boring
Every scheduled job writes a row when it fails: job name, stage, exit code, timestamp. The vocabulary was inherited from an earlier launchd-based version of the same pipeline and kept identical on purpose, so that the health check reading it did not need to change when the jobs moved to GitHub Actions.
