Your team shipped an AI feature. It went out on a Tuesday and it was good.
Six weeks later someone in sales says it has got worse. You ask engineering. Engineering says nothing changed on their side, and they are telling the truth. You ask for numbers and there are numbers, and none of them answer the question. You are now in a meeting about whether a thing is worse, with no way to settle it, and the meeting will happen again next month.
That meeting is what AI evals exist to prevent. And the reason most teams do not have them is not technical.
What an AI eval actually is
Strip the tooling away and an eval is three things.
