Keeping the LLM out of the verdict
I've been building a system that audits AI-generated fitness training plans against published sports-medicine guidelines. A fitness app sends in a plan its LLM produced, and the system returns a verdict: pass, flagged, or rejected — with the specific rule and source it was checked against.
The obvious way to build this is to hand the plan to a good model with a well-written prompt: "Here are the ACSM guidelines. Does this plan comply?" It would work. Most of the time.
I didn't do that, and this series is mostly about why, and what it costs.
The problem with asking a model to be the judge
