Part 3: Build the Eval Set Before the Agent Exists

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch

Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet.

That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo.

The eval set: eval/cases.json

21 cases, three buckets:

Part 3: Build the Eval Set Before the Agent Exists

Related reading

Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

Part 4: The Raw ReAct Loop: ~100 Lines, No Framework

Part 5: Guardrails That Live in Code, Not the Prompt

Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a…

I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story

Part 2: Pinning the Use Case and Writing Tool Contracts Like Specs