I was building an eval harness for a retrieval-augmented generation pipeline, and the first faithfulness check I wrote was quietly wrong. It looked reasonable. It ran on every example for free. It just measured the wrong thing, and I only saw it once I started feeding it edge cases on purpose. The way it fails is the same way most RAG tutorials I had copied from fail.
Here is the one sentence version: token overlap does not measure faithfulness, it measures copy-paste fidelity, and the gap between those two destroys your eval exactly in the cases you care about.
Let me show you what I mean, because "use a better metric" is the kind of advice that sounds smart and helps nobody.
What faithfulness is supposed to mean
Faithfulness is a different axis from relevance. Relevance asks whether the answer addresses the user's question. Faithfulness asks something narrower: is the answer grounded in what the retrieved context actually says? You can be perfectly faithful and useless (you quoted the context exactly but answered nothing the user asked), or relevant and unfaithful (you answered the question correctly using facts you invented). A real eval needs both, measured separately. Collapsing them is the first mistake, but it is not the one I want to talk about.
