If you're using or building on large language models (LLMs), perhaps the most important concept to understand is how inference and key-value (KV) cache work. That's because no matter if you're working with coding agents, retrieval-augmented generation (RAG), or fine-tuning, inference is what happens each time you make a request to a model and get a response back (and thus, where all the money goes).While training happens once, inference happens every single time a user sends a prompt. So, let's walk through how it works, what the KV cache is, and optimizations that most teams use to save on infrastructure costs and reduce latency.Inference is a stack, not a model file
Figure 1: Serving a model takes 3 pieces working together: the weights, an inference server, and the hardware underneath.A model sitting on your machine (or HuggingFace) doesn't serve anybody, yet. For inference to be possible, you need 3 pieces working together:Model weights: The file(s) with the billions of learned parameters: Kimi, GLM, Qwen, or whatever you've picked.Inference server: Software like vLLM that loads the model, manages incoming requests, and handles the optimizations we're about to cover.Hardware accelerator: Usually a GPU, doing the heavy numerical lifting.You can skip the middle layer and run a model straight on a GPU with PyTorch. That works fine for a notebook or a single user. The moment you need to serve many people at once, however, the inference server is what makes the GPU usable at production scale (think about you opening an HTML file locally versus serving it using Apache HTTP server).Models generate 1 token at a time
