Originally published on kuryzhev.cloud
Context
We had a single Helm chart deploying roughly 15 microservices, split across staging and production, with ArgoCD 2.9 syncing everything from a monorepo. The pattern looked textbook: one values.yaml as the base, then a values-staging.yaml and a values-prod.yaml to override whatever was different per environment. On paper, this is the recommended way to manage helm values staging production configs, and honestly, we thought it was a solved problem. Helm 3.14, chart apiVersion v2, nothing exotic.
Our assumption going in was almost embarrassingly naive: "just override values.yaml per environment, how hard can it be." We'd done this a dozen times on smaller projects. What we didn't account for was scale — 15 services, two environments, and a growing team all touching the same values files without a shared mental model of what "override" actually meant in practice.
Three incidents forced us to rethink the whole approach: a silent config drift that caused a production timeout bug, a resource-sizing mistake that both wasted money and OOMKilled a payment service under load, and a security lapse where secrets ended up committed to a values file in git. None of these were exotic failures. They were the kind of thing that happens when "it's just YAML" turns into "nobody actually knows what's in prod anymore."
