RAG solves a real problem: language models hallucinate when they don't have relevant context, and they can't access data beyond their training cutoff. But naive RAG — embed a query, retrieve the top-K chunks, feed them to the model — has a precision problem. The chunks you retrieve are often related to the query, not relevant. Re-ranking fixes this by scoring retrieved passages a second time, with a model that understands semantic relationships at a finer grain.
This article walks through building a RAG pipeline with re-ranking in Python, from embedding to final answer generation.
Why Naive RAG Falls Short
The standard retrieval step uses a bi-encoder: embed the query and all documents into the same vector space, then find the nearest neighbors by cosine similarity. This is fast enough to run at scale, but bi-encoders compress meaning into a fixed-size vector — they can't compare query and passage directly.
The result: you retrieve chunks that are topically similar but not actually useful. Ask "what are the access control requirements for SOC 2?" and you might get chunks that discuss HIPAA or ISO 27001 instead. Nearby in vector space, useless in practice.
