A news platform I maintain started serving pages in six seconds. Load average sat at 16 on a 12-core box for hours. Nothing had been deployed. Traffic was up, but not 10x up.
The cause turned out to be a single SELECT that looked completely reasonable — the kind of query that passes code review, works fine on a 5,000-row table, and quietly becomes a wrecking ball at 80,000 rows.
Here is the whole investigation: how I found it, why it was slow, what the fix was, and the three unrelated things I learned along the way.
Symptom first
The obvious metrics:
