A practical look at the retrieval layer of RAG — what a vector database actually does, why keyword search fails, and what it costs to get it wrong.
A few weeks ago, a health-tech client sent me a screenshot that made me stop typing. His RAG system had answered a doctor's question about a medication interaction with a dosing suggestion that was wrong — not subtly wrong, dangerously wrong. The team's first instinct was to blame the model. I asked one question: "How is your system retrieving the context?" Nobody knew. We opened the code and found it: keyword search over their knowledge base, feeding the top hits into the prompt. There was no vector database anywhere in the pipeline.
That answer explained everything. The model had done its job. It read three irrelevant chunks and produced a confident, grounded-sounding, completely wrong answer. The retrieval was broken — there was no way for it to find meaning, only matching strings.
That incident is why I am writing this. Vector databases are not a trendy accessory you bolt onto an LLM project. They are the difference between a RAG system that quotes the right source and one that confidently quotes the wrong one. This article covers what a vector database actually does under the hood, the taxonomy of options, a production architecture, real code, and the honest failure modes — including when you should not use one at all.## The Problem Keyword Search Can't Solve
