Originally published on hexisteme notes.
No — a scoring function that consumes whatever values an LLM hands it is only deterministic in name; the LLM's judgment launders straight through the "deterministic" gate, and you close the hole with three rules (host-verified FACT sourcing, FACT-only scoring, and an asymmetric penalty where bad signals are penalized regardless of provenance while good signals only score when verified) plus multi-round adversarial testing.
The load-bearing trick is an asymmetric-penalty mechanism: an unverified input can only ever lower a score, never raise it. Bad signals are penalized regardless of where they came from (so you can't dodge a penalty by routing the bad news through a weak source), while good signals are credited only when they carry a FACT provenance. We hardened this through three rounds of adversarial review, with an attacker-satisfaction score climbing 58 → 71 → 96 as each round peeled back a deeper laundering channel — and the decisive case was a candidate that scored 94/ADOPT on 9 web-sourced (unverified) signals plus a single real FACT, which the fixed gate correctly flipped to AVOID.
The trap: "the LLM only extracts, Python decides" is not enough
