The plan output was four hundred lines long. Buried around line two hundred and ten, between a tag change and a security group rule, was a resource marked for replacement. It was the managed database. Somebody approved the plan in about eleven seconds, because the change description said "update instance tags," and it did do that too.
We caught it because the apply took longer than a tag change should and someone looked. We got lucky. The immutable field that triggered the replacement was a subnet group attribute a colleague had changed in a shared module three weeks earlier, and it had been sitting harmlessly in the module until the next plan happened to pick it up. The person who wrote it never saw a plan for this environment. The person who approved the plan never saw the module change.
Reading a plan is a skill nobody teaches and everyone is assumed to have. Four hundred lines of diff, mostly noise from computed attributes that will resolve at apply time, and the one line that matters is formatted exactly like the ones that don't. Humans are bad at this. We are especially bad at it at 5pm on a change we believe is trivial.
So we stopped relying on reading. Plans now go through a policy check before they reach a human. Any destroy or replace of a resource in a protected set fails the pipeline outright and requires a separate, explicitly labelled change with a second approver. Stateful resources carry prevent_destroy lifecycle blocks, which is blunt and occasionally annoying and has now stopped this twice. The plan is saved as a file and the apply runs against that exact plan, so what got approved is what gets executed.
