Author(s): Caden Lippie
Originally published on Towards AI.
Originally introduced in the 2017 paper “Attention is All You Need” (Vaswani et al., 2017), transformers form the structure for most modern large language models, including ChatGPT and Claude. In fact, the GPT in ChatGPT stands for Generative Pre-trained Transformer.
Transformer architecture blends small feed-forward MLPs with attention mechanisms and has been shown “to be superior in quality while being more parallelizable and requiring significantly less time to train” (Vaswani et al., 2017) when compared to previous model designs.
In an attempt to grasp these mechanisms more thoroughly, I decided to build my own transformer and compute an entire forward pass by hand. This builds directly on the feed-forward network I worked through by hand in my MLP Walkthrough, adding attention, positional encoding, and the encoder-decoder structure. Working through every matrix multiplication myself gave me a much more confident understanding of how these pieces actually fit together, and made a technology that can otherwise feel like a black box a lot less mysterious.
