The reranker I added to improve RAG was causing most of my remaining misses

My RAG evaluation would tell me hybrid: 0.86 and I'd sit there with no idea what to actually change. Raise it how? Different embedder? Smaller chunks? Add a reranker? The aggregate score names a winner; it says nothing about why the losers lost.

So I built a small tool to answer the "why," and running it turned up something I didn't expect: in my best-performing configurations, the component I'd added to improve results — the reranker — was responsible for almost every remaining failure.

Here's the idea, the finding, and how to check it on your own corpus.

The problem with a single retrieval score

hit@k = 0.86 is an average over a lot of independent failures, and those failures don't all have the same cause. A query can miss because:

The reranker I added to improve RAG was causing most of my remaining misses

Related reading

Top Reranking Models to Boost RAG Accuracy in 2026

RAG Rerank: the Highest-Leverage Upgrade to Your Retrieval Pipeline

How Do You Know Your RAG Actually Works?

Your RAG Retrieved the Right Documents but Still Gave the Wrong Answer

5 Failure Modes I Found in My Financial RAG (And the One That Actually Mattered)

Most RAG Problems Are Retrieval Problems. Here Are 8 Fixes That Worked for Me