A common failure pattern in a retrieval-augmented generation (RAG) system is a progressive decline in performance. This decline, which can be difficult for users to detect initially, often begins with a reduction in retrieval relevance. Over time, it may lead to longer response times and increasingly inaccurate, incomplete, or less helpful responses. This gradual degradation of the system's performance creates a challenging user experience.
Production failures often stem from uncoordinated changes, with operators adjusting retrieval settings, reranking methods, or model routing without a shared change process. Without explicit versioning and ownership, it becomes difficult to trace which change caused a regression or who made it.
This article argues that production AI pipelines, particularly RAG systems, must be designed around explicit control of change. The system must treat retrieval and prompting, evaluation, and model selection as controllable elements that people running the system must be able to modify through visible changes during active system use. The goal is not to introduce new techniques but to show how existing, well-understood methods can be composed into a production system that remains stable, measurable, and adaptable over time.
