I gave the same fabricated answer to RAGAS and DeepEval. One scored it 0.0. The other scored it 1.0

Here's an output from a RAG system asserting a pricing claim it was never given, for a question its context couldn't answer. I ran it past the two most popular LLM-as-judge faithfulness metrics, five times each, judges on gpt-4o at temperature 0 — the setting most favourable to judge stability.

RAGAS scored it 0.000.

DeepEval scored it 1.000 — in all five repeats — and explained itself: "The score is 1.00 because there are no contradictions between the actual output and the retrieval context." One repeat added: "Great job maintaining accuracy and consistency!"

Both metrics are called faithfulness. Both are internally consistent. Only one of them notices the fabrication.

I want to explain why that happens, what else I found when I measured it properly, and why my conclusion is use both — an LLM judge and deterministic checks — rather than the takedown of LLM-as-judge you might be expecting.

I gave the same fabricated answer to RAGAS and DeepEval. One scored it 0.0. The other scored it 1.0

Related reading

LLM-as-a-Judge: I Built One From Scratch, Then Checked It Against Humans

A 150M model that beats GPT-4-as-judge at catching RAG hallucinations trained…

A RAG evaluator that admits what it can't judge

Your LLM-as-judge disagrees with itself between runs

I checked six LLM-as-judge tools against human labels. The scoreboard was the…

LLM-as-judge disagrees with itself between runs