The service was slow. Not down, just slow: p95 latency climbing well past where users notice, requests piling up, the kind of degradation that generates support tickets instead of alerts. And the autoscaler, the whole point of which is to add capacity when a service is under strain, sat there doing nothing.
The metric it was watching said everything was fine. Average CPU utilization on the tasks was hovering around 30 percent, nowhere near the scale-out threshold. The dashboard was calm. The users were not. Both were right, and the gap between them is one of the most common autoscaling traps on a container platform.
This is the first article in a series on running a multi-tenant SaaS on AWS at team scale. It is about a metric that lies, quietly, by design.
Why 30 percent CPU meant 100 percent busy
The service was a single-threaded application. A Node.js API, in this case, but the same is true of any process that does its real work on one thread: a classic Python or Ruby worker, most single-process runtimes.
