I watched a customer demo their new RAG application last month. They had spent three months building it. The interface was beautiful. The LLM responses were eloquent and well-formatted. And almost every answer was subtly, confidently wrong.
“The model keeps hallucinating,” they told me, frustrated. They had tried three different LLMs. They had adjusted temperature settings. They had rewritten their prompts a dozen times. But here is what nobody had looked at yet: the search layer feeding context into those prompts.
This is the part of RAG that deserves more attention than it gets: most quality problems are retrieval problems. When your generative AI makes things up, it is usually because the context it received was off. Too much, too little, or just plain irrelevant. The LLM is doing exactly what you asked: generating fluent text based on the information you provided. If that information misses the mark, well, garbage in, eloquent garbage out.
The challenge is that retrieval tends to get treated as a solved problem. Chunk the documents, throw everything into a vector database, retrieve the top 10 results, done. That works for demos. In production, though, you need the system to distinguish between a precise question about part number XJ-447 and an abstract question about how plumbing works. That distinction is where things get genuinely interesting.
