A model that only ever read text now looks at a photo of your fridge and suggests a recipe. Nothing about "predicting the next token" obviously explains how it can see. The bridge is simpler and more elegant than it looks.
Multimodal AI — models that handle images, audio, and text together — feels like a different kind of technology. It isn't. It's the same machinery with one clever addition. Here's the idea.
The core trick: everything becomes the same kind of thing
A language model reasons over embeddings — vectors that capture meaning — derived from text tokens. The entire breakthrough of multimodal models is this: you can turn an image (or audio) into vectors in that same space.
An image gets sliced into patches, and each patch is encoded into a vector, much as a sentence is split into tokens and embedded. Once an image is a sequence of vectors living in a shared space with text, the model's attention mechanism can treat words and image-patches as one combined sequence — letting a word attend to a region of an image and vice versa. "See" isn't a new faculty bolted on; it's the old faculty fed a new kind of input that's been translated into the language the model already speaks: vectors.
