CloudWatch alarms are easy to mark as "done" long before the notification path is actually safe. The metric flips, the alarm state changes, and the Terraform plan applied cleanly. Then the first real incident lands in an old team alias, or it arrives with the wrong enviroment name in the subject, and nobody trusts the alerts anymore.
I started treating alarm email delivery as part of the release gate, not a side check for later. If the pipeline only proves that AWS resources exist, you are validating configuration, not the operator experience. Those are not the same thing, and the gap is where messy on-call surprises tend to start.
Why CloudWatch alarm email tests keep giving false confidence
The common failure pattern is boring:
an SNS topic exists, but the subscription points at the wrong mailbox






