When you optimize the inference performance of an LLM, you need to know how to measure it. Without measurement, it is easy to make a model more complicated without making it faster, or to improve throughput while making user-visible latency worse.
An LLM service has several kinds of performance. A user cares about how long it takes to see the first token and how quickly the rest of the answer streams. An operator cares about how many requests the hardware can serve, how much memory is used, and how much each generated token costs. A researcher may care about whether an optimization changes the model’s output quality.
In this chapter, you will learn about:
Latency and throughput metrics
Time to first token and time per output token
