I built an LLM eval framework from scratch. Here is what I wish I had bought instead.

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever.

Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?"

The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again.

The one line I wish someone had told me: build the rubric, buy the runner

Here is the split that took me six months to see.