Your agent eval suite grades one turn at a time. Prompt in, output out, score it, move on. That model is fine for a completion endpoint. It is quietly wrong for anything that holds a conversation, and it is the reason your "94% pass rate" coexists with users who rage-quit on turn six.
The unit of failure for a conversational agent is not the turn. It is the session.
The per-turn blind spot
Consider a support agent. Turn 1: user asks about a refund. The agent answers correctly and passes every eval. Turn 4: user clarifies they mean a partial refund. The agent answers that correctly too. Turn 6: the agent quotes the original refund amount again, having silently dropped the "partial" constraint three turns ago.
Every single turn passes an isolated eval. Each output is grounded, well-formatted, on-topic. The failure only exists across turns: a dropped constraint, a contradiction with turn 4, a promise never kept. Per-turn grading is structurally blind to it, because the bug is in the relationship between outputs, not in any one of them.
