Why the memory column on an EC2 GPU instance is not the number you need.
I have run AWS infrastructure for years. When I started looking at how large language models actually get served, I hit a wall on something basic.
Someone said "the model sits on the GPU." Someone else said "the GPU does the calculation." Both sounded right. Neither told me where the model actually lives.
The problem is that "GPU" is not one thing.
Two parts, one board
You already know how a normal machine is laid out. A CPU does the work. RAM holds the data. They sit apart, connected by a bus.
Which hardware should run this model? Most teams answer that question with habit instead of arithmetic. The model card says GPU.…
Why a model that "fits" in your GPU still OOMs — the KV cache, overhead, and quantization math self-hosting tutorials leave out.
aws.amazon.comIf you’re iterating on deploying large language models (LLMs) on AWS GPU instances, you’ve probably noticed the larger the model…
aws.amazon.com
What engineers actually consider when choosing GPU infrastructure, beyond hourly price and VRAM.
Gemma 4 E2B on the two cheapest whole-GPU CUDA instances AWS sells. Same Turing GPU, different host CPU. One is cheaper per hour,…
I run this homelab on two small boxes. One never turns off and just serves DNS and background services; the other has a GPU with…
