Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory

Here is the short version: the model weights are the smallest GPU-memory surprise you'll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most "run an LLM on your GPU" tutorials load the weights, run one short prompt, and declare victory. Then you send a 6,000-token document at a batch of eight and it OOMs.

I've set this up enough times to know the failure isn't random. It's arithmetic you can do before you rent the GPU. This post is that arithmetic.

Why does a model that "fits" still run out of memory?

GPU memory for inference is four separate buckets, and tutorials only mention the first:

Model weights — fixed, predictable, load once.

Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory

Related reading

Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host…

Admission Control for Self-Hosted LLMs: Rejecting Requests Before the KV Cache…

KV Cache Is Eating Your VRAM — Here's How to Estimate It Before You Run Out

8GB to 70B: A Real Hardware Guide for Local LLMs

How Much RAM Do You Really Need to Run LLMs Locally? 2026 Benchmarks

Why Your Local LLM Setup Is Costing More Than You Think — And What Happens When…