AI agents are non-deterministic. This is both their superpower and their biggest testing challenge.
When you change a prompt, swap a model, or update a tool definition, the agent's behavior shifts in ways you can't predict. Traditional test tools (Jest, Vitest, Playwright) are built for deterministic code — they expect exact string matching. AI agents don't work that way.
Last week, I changed a system prompt in a support agent. It still passed all our Jest tests (the function still returned a string). But the agent had stopped offering password reset help — a regression that only surfaced when a user complained.
That's why I built AgentSpec.
What is AgentSpec?
