Building an Agent Evaluation Harness for local AI agents. Architecture, code, and results from testing a Nifty trading agent. Key findings: overfitting detection, precision/recall tradeoffs, expiry week weaknesses. 5 surprises from 300 test cases. Why evaluation-driven development matters for AI agents.
Evaluation harness tests AI agents on 300 scenarios: accuracy, precision, F1, latency, cost, hallucination rate vs. baselines. Most skip formal evaluation and confuse manual overrides with real performance; systematic testing prevents shipping untested agents to production.
I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong
DOYR | Not financial/legal/tax advice. For educational purposes only.
Three months ago, I started building AI agents for my trading business.
First agent: Fetches Nifty option chain data.
Second agent: Analyzes PCR, OI, max pain.
Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument:...
You have an agent that works. Now someone asks how you know, and the honest answer is that you tried...
The agent passed every test I threw at it by hand. Then a user asked it to "book the cheaper flight,"...
I gave the free tier a cheaper model and it invented conference speakers who don't exist. What that taught me about model…
Build an AI agent evaluation harness with task fixtures, trace scoring, judge checks, regression tests, budgets, and human review…
I run a one-person AI company. A few weeks ago I pointed Claude Code — an AI coding agent — at a...
