If you have spent time understanding how LLMs process text, you have probably come across Byte Pair Encoding. It is the algorithm sitting quietly under the hood of GPT, Llama, Mistral, and most other major models, turning raw text into a sequence of tokens before anything else happens.
The algorithm itself is elegant. Start with a vocabulary of all 256 individual bytes. Find the most frequently occurring adjacent pair of tokens in your training text, merge it into a new single token, and repeat until you hit your target vocabulary size. That is it. The vocabulary that comes out is entirely data-driven, efficient for the corpus it was trained on, and remarkably simple to implement.
For anyone who wants to understand this from first principles, Andrej Karpathy's minbpe is the obvious starting point. The entire codebase is short enough to read in an afternoon, thoroughly commented, and covers everything you need: a basic BPE tokenizer, a regex-based variant that mirrors how GPT-2 and GPT-4 handle text preprocessing, and even a wrapper that reproduces GPT-4 tokenization exactly. You can train a tokenizer, encode text, decode tokens, save and load. It does what it says.
The problem is not what minbpe does. It is what happens when you try to use it on anything real.
