AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score is stable enough for CI.
LLM judges can be useful for genuinely semantic questions. They are a poor default for verifying tool order, retry limits, validation, budgets, state transitions, or error handling. Those behaviors have explicit contracts and can usually be tested with ordinary deterministic techniques.
The most effective agent test suite separates orchestration correctness from language quality. Test the first on every commit with fakes, traces, and rules. Evaluate the second with a smaller, calibrated model-and-human workflow.
What Can Be Deterministic?
An agent may produce variable language while still following a predictable execution contract. Useful deterministic checks include:
