One seemingly innocent parameter adjustment — increasing group_concat_max_len to accommodate a business requirement — caused a cascade of performance degradation across a GBase 8a production cluster. A simple TOP‑N query that normally completed in seconds suddenly ran for over three hours, and multiple other queries on the same node slowed to a crawl. This article reconstructs the full investigation, from identifying the bottlenecked node to uncovering the hidden chain that turned a 200,000‑row sort into a 10 TB disk write storm.
1. Symptom: One Node’s I/O Pegged at 100%
Monitoring showed several queries exceeding 10,000 seconds of execution time. Cross‑referencing the coordinator‑level task view and the data‑node task view revealed that all slow queries were pinned to node3.
-- Coordinators
SELECT COORDINATOR_NAME, ID, user, host, command, start_time, time, state,
