When a Vietnamese bank's internal AI assistant started confidently quoting compliance rules that did not exist in any document, the team discovered they had been testing the wrong thing entirely. This post walks through how we set up RAGAs evaluation on that project, what faithfulness, context recall, and answer relevance each actually measure, and how the three metrics together gave us a diagnostic that the "looks good in demo" approach never could.
Key takeaways
Faithfulness catches the generation layer hallucinating facts that contradict the retrieved context — a score below 0.85 in production means confident wrong answers are reaching users.
Context recall is a retrieval-layer metric: a faithfulness score of 1.0 combined with low context recall means the system is accurately summarizing incomplete information, which is just as dangerous.
Answer relevance penalises over-hedged, padded responses that technically address the question but bury the useful content — high faithfulness and recall can still coexist with low relevance.
