In classical test-driven development, we deal with deterministic outcomes. We write assertions against values and types we already know in advance — assert result == expected, and we move on.
AI agents don't play by those rules. Because the underlying LLM is non-deterministic, giving it the same input twice can produce two differently-worded outputs. That alone breaks the classical assertion model.
But there's a second problem, beyond just consistency. Say we have a customer support agent, and we want it to be helpful, empathetic, professional, and friendly when it talks to customers. Even if the output were consistent, how do we test for qualities like that? There's no fixed value or type to assert against — "empathetic" isn't a type or an exact string, it's a subjective judgment call.
So we're dealing with two distinct problems: unpredictable output, and qualities that are inherently subjective. Testing an AI agent means solving for both.
Enter Pydantic Evals
