A context window is the maximum number of tokens a language model can process in one go. It includes the system prompt, conversation history, and the model's own output. Inside the model, self-attention and a key-value cache enforce this limit, acting as the model's working memory. When the window fills up, older information falls out, and the model can no longer use it.
But here is the twist: the model's performance drops even when the window is not full. If you bury a crucial detail in the middle of a long prompt, the model often ignores it. And in a long chat, the assistant might start hallucinating long before you hit the advertised token limit.
In the last episode, we saw how training and inference are separate phases. Now we will look at the inference-time machinery that decides how much of your conversation the model can actually keep in mind.
What exactly is a context window?
Imagine you are reading a long recipe on a small phone screen. You can only see a few lines at a time. To follow the recipe, you scroll up and down, but you cannot see the whole thing at once. The visible area is your context window.
