I wanted to dive deeper and learn more about the underlying architecture and function of LLMs. I discovered Andrej Karpathy's YouTube video, "Deep Dive into LLMs like ChatGPT" to be extremely informative and also full of resources to keep learning on my own.
As a software engineer, I use Claude Code to write code, generate architecture diagrams, investigate issues, complete Jira tickets, unit test, and for basically all functions of my engineering role. I use ChatGPT to dive deeper into concepts, research topics and improve my writing. But how do these models get to be so smart? How can they be optimized for a specific task, for example coding? What is improved when a new model is released? I wanted to understand more about the inner workings of an LLM so I can make the most of my use of them.
Here are some things I learned from this video:
The difference between a base model and a chat or instruct model. I learned that fundamentally, base models are stochastic generators of text. The model does not have knowledge of self or function as a chat assistant. It's essentially an autocomplete engine from the internet dataset it was trained on because it's not optimized to interact with humans or provide assistant-like output that we expect in products like ChatGPT.
