Why Fewer, Better-Chosen Chunks Beat a Bigger, More Expensive Model
For a while, my retrieval setting was top_k = 10. It felt responsible. If I only pulled back 3 chunks and the right one wasn't among them, the answer would be wrong for a reason that had nothing to do with the model. Pulling back 10 felt like insurance — cast a wide net, let the LLM sort out what actually mattered.
The bot's answers didn't get better. Some of them got worse.
I remember one question in particular — something specific, with a clear correct chunk sitting somewhere in the middle of those 10. The LLM's answer wandered. It grabbed a detail from a chunk that was only loosely related, ignored the one that actually answered the question, and produced something that read confidently but missed the point. The right information was in the prompt. It just wasn't the information the model reached for.
That's when I went back and actually read what I'd been sending. Ten chunks, most of them tangential, all competing for the model's attention in one long prompt. The correct answer wasn't easy to find in there — even for me, reading it slowly, on purpose. I'd been treating "send more context" as a safety net. It was actually working against the model, and against my bill, on every single request.
