How to evaluate a RAG system: recall, faithfulness and the questions that matter

You cannot improve a RAG system you are not measuring, and "I asked it five questions and it seemed fine" is not measuring. Every serious change I made to RAG.NextUpgrad — hybrid search, reranking, the confidence gate — was justified by a number moving on a fixed evaluation set. Here is the setup, sized for a student or a small team.

Build the question set first

Collect 50 to 100 real questions. For each, record the passage (and page) that answers it, and the expected answer in one sentence. Include 10 to 20 questions the documents cannot answer — those test whether the system refuses correctly. This set is your most valuable asset; version it alongside the code.

Four metrics, in order of importance

Retrieval recall@k — is the correct passage among the top k retrieved? If this is low, nothing downstream can save you. Fix chunking and retrieval before anything else.

How to evaluate a RAG system: recall, faithfulness and the questions that matter

Related reading

RAG Evaluation: How to Know Your Retrieval Isn't Broken

Eval-First RAG: Use Separate Scores to Triage Failures

RAG Evaluation Checklist for AI SaaS: Catch Bad Answers Before Users Do

How Do You Actually Evaluate Your RAG App?

Why RAG gives wrong answers (and how to fix retrieval failures)

Your RAG System Is Lying To You About That Table