Originally published on kuryzhev.cloud
One bad helm upgrade and your production values file is history — literally — unless you have a helm rollback strategy that survives concurrent deploys, stateful side effects, and plain human error. I've watched a team roll back to what they *thought* was the last good revision, only to land on a "superseded" release with a broken feature flag still baked in. Here's what we changed after that incident, and what we now enforce before every promotion.
Pin exact chart versions and snapshot values before every promotion
The biggest source of rollback confusion is not knowing what was actually deployed. values.yaml in your chart repo is not the source of truth — the live cluster state can drift from it after hotfixes, manual --set overrides, or an emergency patch someone forgot to commit. Before every helm upgrade, run helm get values <release> -o yaml and store it alongside your CI artifacts, tagged with the git commit SHA and the release revision number. If you can't answer "what values were live five minutes ago" in under 30 seconds, your rollback plan is theoretical, not real.
Roll back by revision number, never by "previous" assumptions
