TL;DR
Our checkout API's p99 latency crept up to 2.1s and nobody could pin down why. I paired with Claude Code to profile it, found two dumb mistakes hiding behind "it's probably the database," and cut p99 to under 900ms in an afternoon. This is the actual debugging session, not a cleaned-up retelling.
The Problem
A few weeks ago our checkout API started showing up in the "slow endpoints" dashboard. Not down, not erroring — just slow. p50 was fine at 180ms, but p99 had drifted to 2.1 seconds over about a month. Nobody noticed until a support ticket mentioned "checkout feels laggy sometimes."
That "sometimes" is the annoying part. Intermittent tail latency is one of the worst debugging problems in software because:
