TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one question a merge queue cares about: does this gate pass or fail the same way twice. I wired six of these frameworks into a real GitHub Actions merge queue and ran them against production PRs for about eight months. The ones that gate cleanly share a single property: deterministic checks that return an exit code in seconds, with LLM-as-judge scores kept as non-blocking signals. The ones that flake share the opposite: nearly every metric is a judge call, so the queue blocks on a number that drifts. Ranked by "survived our merge queue," Promptfoo and DeepEval came out ahead. The short list first, then per-tool notes, then when you should not gate on any of them.
The outage that set the ranking
Two years ago I put an LLM-as-judge metric on our merge queue with a 0.8 threshold. It looked clean in the demo. Three weeks later it blocked fourteen PRs and a release over a weekend, because the judge scored the same unchanged output 0.83 on Friday and 0.78 on Monday. Same prompt. Same model. No seed. I killed the gate at 1am from my phone and we shipped fine. The regression it was "protecting" us from never existed.
