Author(s): GSO1
Originally published on Towards AI.
Introduction
Picture an input prompt to a large language model as a cloud of points in a high-dimensional vector space E, one point for each token. As the model processes the prompt layer by layer, the cloud reconfigures itself — points shift position, some drift together, others pull apart — so that each token’s location better reflects how it relates to the others in context. By the final layer, the cloud has settled into a configuration that encodes the contextualized meaning of the prompt. This final configuration is not a random scatter: semantically related tokens have drifted into clusters, a structure we will see emerge as the natural endpoint of the dynamics described in this article. The next token is then read off from a single point in this configuration: the last token’s final vector determines a set of alignments with the rows of the unembedding matrix and softmax over these alignments produces a probability distribution over the vocabulary, from which the next token is sampled.
This article is about that reconfiguration — what drives it, what shape it takes, and what it reveals about transformer attention.
