I've Built RAG Infrastructure Several Times. Last Week Was the First Time I Actually Benchmarked It.

I have a confession, and I suspect I'm not alone in it: I've built RAG infrastructure multiple times, and until last week I had never benchmarked any of it.

Unit tests, sure. Integration tests, sure. Everything green, every pipeline connected. But if you'd asked me "is the retrieval actually good?", the honest answer was a shrug with a deployment attached.

For context: I'm building Langhuan, an open-source knowledge base that agents query over MCP. Retrieval is the product. Which made the shrug harder to live with.

The audit that started it

Before writing any eval code, I sat down and listed the retrieval decisions I had shipped. The chunking contract had been through three revisions. Vector and keyword search get fused with RRF. A reranker sits on top. Every one of those decisions had a rationale I could defend in a code review. Not one of them had a number behind it.