Gating an agent's eval run by severity means the deploy decision reads the per-severity distribution of the failures, not one flat pass-rate. severity_gate.py is an offline, stdlib-only tool that returns SHIP, REVIEW, or BLOCK. In this post's fixtures, flipping one severity_class field to critical turns SHIP into BLOCK while the pass-rate holds at 92.5%.
AI disclosure: I wrote severity_gate.py with an AI assistant and ran it myself, offline, before publishing. Every output block below is pasted from a real local run on Python 3.13.5, standard library only, no network. I checked the exit codes (0 / 1 / 2), hashed the STDOUT of each scenario twice to confirm it is byte-for-byte deterministic, and edited every line. The 512 / 31 / 6 figures and the "rounding error" line belong to Ethan Walker's Dev.to post, not to me; I link the primary sources and keep their numbers in their own paragraphs, away from my fixture counts (40 / 3 / 2, 92.5%).
In short:
Whether a run is safe to ship is a property of which failures it holds, not of their average. A run can sit at a healthy pass-rate and still carry the one failure that should have stopped the deploy on its own.
The tool reads a finished eval run (JSON) plus a small policy. Any FAIL in a blocking severity class (default: critical) is a BLOCK, whatever the aggregate says. A below-threshold aggregate with no critical fail is a REVIEW. Everything clean is a SHIP.
