Testing AI agents is hard. I built a framework for it.

Your AI agent works in dev. You change a prompt to improve tone. Now it stops routing billing questions correctly.

You don't find out until a user complains.

The problem: AI agents are non-deterministic. Traditional unit tests don't work. expect(output).toBe("transfer to billing") fails 30% of the time on correct behavior, and passes when the agent is broken in subtle ways.

I built AgentSpec to fix this — a testing framework designed specifically for non-deterministic AI output.

What it looks like