GPU capacity is the tightest it has been in a while. Amazon just crossed three trillion dollars largely on cloud AI demand and the reporting says even AWS can't add capacity fast enough. When supply is that tight, the price gap between on-demand and spot GPUs gets wide and interesting, and every team running inference is staring at the same question: how much of this can we safely move to spot?
I'm a cloud associate and this is a decision I've had to make with real money attached, so instead of the usual "spot is 70% cheaper, go use it" take, here are the actual rules we use to decide what goes where. Spoiler: the headline discount is the least important number.
First, the number everyone quotes and why it's misleading
Yes, spot GPU instances often run 60-70% below on-demand. That number is real and it's also a trap, because it quietly assumes your workload doesn't care about being interrupted. GPU spot capacity is the first thing reclaimed when demand spikes, which right now is often. So the real comparison isn't "cheap vs expensive," it's "cheap-but-can-vanish vs expensive-but-guaranteed," and the right answer depends entirely on what the workload does when it gets a two-minute eviction notice.
