The workflow had been running for six hours when the first alert fired. The apply stage was exiting with a JSONDecodeError, the supervisor was restarting it, and every restart died on the same line of the same file. The queue showed zero unacked messages, which meant the work was already marked done. I had a crash loop that would not die, and a set of decisions that were recorded but never applied.
My first instinct was to blame the model, and that instinct was wrong. This is the story of how the real root cause turned out to be a boundary condition in the pipeline, and why a disposable server and a free token allowance turned a panicked debugging session into a reproducible experiment.
The symptom
Here is the failure signature in its simplest form. The apply stage read the ledger line by line, parsed each line as JSON, and applied the decision to the database. One line was truncated mid-object, json.loads raised, and the process exited with code 1. The supervisor saw a non-zero exit and restarted the stage, which re-read the ledger from the top and hit the same line again.
writer -> ledger.jsonl: append raw model output (truncated)
