Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval *and* Honesty

Part 4 of Retrieval-Augmented Self-Recall. Code: RE-call. Part 3: the honesty guards.

A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.

That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: when there was an answer, did you rank it first? None of them ask the question that decides whether agent memory is safe: when there was no answer, did you say so?

So RE-call ships its own harness. Here's how it works, and the first finding it produced.

The test set: the unanswerable queries are the point

Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval and Honesty

Related reading

Retrieval-Augmented Self-Recall — Part 3: Teaching RAG to Say \"I Don't Know\

Retrieval-Augmented Self-Recall: The RAG Problem Nobody Talks About

Retrieval-Augmented Self-Recall — Part 5: The Gap Threshold That Didn't Transfer

Retrieval-Augmented Self-Recall — Part 6: The Fine-Tune That Did Nothing, and…

Retrieval-Augmented Self-Recall — What the Comments Taught Me (RE-call v0.3)

Retrieval-Augmented Self-Recall — Part 2: Hybrid RAG on Nothing but Postgres