The metric taxonomy for RAG — retrieval metrics, generation metrics, the eval sets you build, and the numbers that tell you which layer is failing.
I was reviewing a team's RAG demo at a meetup last year. The presenter asked a question, the system retrieved a chunk, and the LLM produced a fluent, well-cited answer. It looked perfect. I asked the question nobody in the room could answer: "What is your hit rate?" The presenter blinked. "What do you mean, hit rate?" We checked their logs. The retrieval was returning the correct chunk about 30% of the time — and because the demo questions were chosen for answers the system could find, nobody had noticed.
That is the whole story of RAG evaluation in one incident: fluent output hides broken retrieval. The LLM is a confident text generator, so a wrong chunk in the context produces an answer that sounds exactly as good as a right one. The only way to know which is which is to measure the layers independently, with metrics, on data you control.
I have written at length about why retrieval quality matters more than model choice. This article is the how: the metric taxonomy, the eval sets you build, the tooling that runs the numbers, and the honest rules for when this evaluation is worth the effort.
