Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument: with agents, you are not just judging an answer. You are judging a run. The path matters. The tools matter. The safety boundaries matter. I ended it by saying I would share the repo when it was ready.
It is ready now. AgentEval Forge is public, on PyPI, and this launch is really a report on what I learned while trying to build a believable way to validate agents.
I thought I was mostly building a scoring system. Real agents turned into the tsunami I was not prepared for, and the project became an integration reality check much faster than I expected.
What I Actually Built
I did not want to build a thin wrapper around an existing eval framework and call it a launch. So I went deep.
