You just shipped an agent. It works in the demo. Now someone asks the reasonable question: "How do we know it keeps working?" And you reach for evals — and hit a wall. You have no labeled dataset. No golden outputs. No historical traces. Nothing to grade against.
So the team stalls. "We'll add evals once we collect data." Meanwhile the agent runs in production, ungated, and the first time it silently breaks is the first time anyone notices.
This is the cold-start problem, and the usual response — "just get an LLM to score the output 1-10" — is exactly the wrong instinct. You do not need labels to start gating. You need to understand which evidence you can trust on day one, and which you can't trust ever.
Independence, not cost
Most eval discussions rank checks on a cost axis: cheap string matches at the bottom, expensive model-as-judge at the top, as if spending more buys you more truth. That's backwards. The axis that matters is independence: can the agent forge this signal, or not?
