The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight," it happily called book on the wrong flight ID, and nobody caught it for three days because the demo I kept running never once asked for the cheaper flight.
That is the trap. When you test an agent by running it yourself, the demo is the test — you type the happy-path prompt, watch it work, and ship. But your users don't type your happy path, and your model provider silently ships a new checkpoint next Tuesday. An AI agent evaluation harness is the thing that tells you your agent actually works, and — more importantly — that last week's prompt tweak didn't quietly break the flight-booking flow you're not looking at.
This tutorial builds one. By the end you'll have a small, runnable Python repo that grades an agent on real tasks, checks that it called the right tools with the right arguments, judges the fuzzy stuff with an LLM, and fails your CI build when a change makes the agent worse. No product pitch, no framework lock-in — the pattern works with LangChain, LangGraph, raw MCP, or whatever you wired together yourself.
What You'll Build: A Minimal AI Agent Evaluation Harness
We'll test a tiny travel-support agent with three tools — search, get_weather, and book — because tool calls are checkable in a way that free-form chat isn't. Along the way we'll build four graders. I call them the Four Graders, and they map cleanly onto what you're actually afraid of:
