You know about LLMs (Large Language Models), but how are they created? Let's build our own to find out!
What You'll Make
By the end of this guide, you'll have trained your very own working LLM from scratch on Shakespeare's complete works (about 1MB of text). The model will learn character-level patterns and generate text that sounds like Shakespeare, not particularly coherent, but with a similarish rhythm and style. The whole process takes about 15 minutes.
Our goal isn't to end up with anything genuinely useful. Big AI vendors spend millions of dollars and months of compute time to achieve that. Our goal is to step through the process at a scale that fits on a typical consumer-grade desktop computer. As we go, we'll learn by doing the same basic steps that every LLM creator (from tiny to mammoth) follows.
Note: This guide was written for the author's MacBook Pro M4 Pro with 24 GB RAM. You may need to adjust specific settings to match your own hardware.
