One afternoon while I was on-call, I received an alert that one of our team's Tier 2 payment services hadn't received any traffic for the past hour. I logged into the Kubernetes dashboard to investigate and immediately noticed that every pod backing the service was stuck in a continuous restart cycle. At the same time, CPU and memory utilization had spiked across every container.
To mitigate the initial impact, I contacted the platform team and looking at the memory spike they decided to provision more memory for the containers. The containers restarted successfully, and the service was restored within 15 minutes.
The Investigation
Although it looked like the memory spike had caused the outage, something didn't add up. This service had never exhibited memory pressure before, and its traffic that day was consistent with its normal daily pattern. I reviewed the Datadog dashboards, looking for anything that could explain the sudden increase in memory usage like an unexpected traffic surge, a recent deployment, or a JVM leak but found nothing unusual.
While increasing the memory allocation had restored the service, it felt more like a mitigation than a root-cause fix. I couldn't justify submitting a code change to permanently increase the service's memory limits in its Helm chart simply to match the temporary configuration the platform team had applied through GitOps during the incident. Before making that change, I wanted to understand why the service suddenly needed more memory in the first place.
