I built a small tool to answer one question: how many GPU-hours is a cluster paying for and not using?
Then I pointed it at real production clusters. Not one of them could answer the question out of the box and the reasons are the same everywhere, because they're all defaults nobody changes.
Here's what I found, in the order it bit me.
1. The utilization metric everyone reports doesn't mean what you think
Almost every GPU dashboard is built on DCGM_FI_DEV_GPU_UTIL. It's the obvious choice it's called utilization, it's a percentage, it goes up when things get busy.
