Multimodal Transformers: How LLMs Learn to See

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A language model can write Python, explain quantum mechanics, and imitate Shakespeare.

Show it a screenshot of a production dashboard, however, and suddenly the central question becomes:

How does a transformer that was trained on text learn what a pixel means?

The naïve answer is: “Give the image to the LLM.”

Multimodal Transformers: How LLMs Learn to See

Related reading

Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Loss Landscapes of LLMs: The Map Beneath Gradient Descent

BPE-Style Tokenizers: The Small Algorithm That Decides What an LLM Can See

Attention Mathematics: Encoder-Only vs Decoder-Only vs Encoder-Decoder LLMs

Tree of Thoughts and MCTS for LLMs: What Happens When You Stop Making the Model…

Reward Hacking in LLMs: When the Model Learns to Win the Game Instead of Doing…