Tokenization is the process of breaking text into smaller units called tokens, the pieces a system actually processes. For a language model, tokens are usually sub-word fragments that the model reads as numbers rather than whole words. In classic search, tokenization instead splits text into the terms used to build and query an index. Either way, it is the step that turns raw text into units a system can work with.
In short: tokenization splits text into tokens, which are the sub-word pieces that an LLM processes, or the terms a search index matches on.
How tokenization works
For language models, common approaches use sub-word tokenization:
Choose a tokenizer: A model uses a specific tokenization scheme. Common approaches include byte pair encoding (BPE), WordPiece, and SentencePiece variants, which all build a fixed vocabulary from sub-word units.
