I built a RAG assistant, then found out my architecture change made it worse

I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened

I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60.

But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out.

The setup

The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic (answer, sources, confidence), using gpt-4o-mini at temperature=0, because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here.