Every time you send a prompt to an LLM, your text gets chopped into tokens before anything else happens. Tokens are discrete integer IDs that the model uses to look up the vectors it actually processes, and that conversion step directly affects how much you pay, how fast your app responds, and how much context you can fit into a single request.Tokenization is easy to overlook until it shows up in your bill or your context window runs out. But if you're building apps on top of LLMs, understanding how that conversion works gives you real control over cost and performance. This guide covers how tokenization works, how tokens relate to vector embeddings, and ways to reduce its impact on your app's speed and budget.What is tokenization? The text-to-numbers pipelineLLMs don't process raw text. They work with tokens, common character sequences drawn from a fixed vocabulary, and learn statistical relationships between them. Your input gets split into these tokens — sometimes at word boundaries, sometimes mid-word, depending on the tokenizer.A useful developer approximation: about 1 token per 4 characters, or roughly three-quarters of a word. The sentence "Hello, how are you?" takes up more tokens than you'd guess by counting words, because punctuation and spacing get their own tokens too.A typical tokenization pipeline has four stages (some implementations fuse or reorder these, but the concepts are consistent):Pre-tokenization: Split raw text on whitespace and punctuation rules. "Hello, world" becomes ["Hello", ",", "world"].Subword segmentation: Apply learned merge rules from the model's vocabulary. Common words stay whole, but longer or rarer words get broken into pieces. "Transformers" becomes ["Transform", "ers"].Vocabulary lookup: Each piece maps to an integer ID. ["Transform", "ers"] becomes [9602, 364].Vector embedding lookup: Those integer IDs get converted into dense float vectors that the transformer actually processes.These four stages are the full path from human-readable text to the vectors the model consumes.This pipeline also creates an important limitation worth understanding early. Because tokenization happens before the model sees anything, the model operates on token-sized units rather than individual characters. A word like "strawberry" may be a single token in many tokenizers, which means the model has no built-in way to inspect its individual letters. This is one cause token-based processing can contribute to failures on tasks like character counting, spelling, and some arithmetic.There's also a hard coupling between a tokenizer and its model. Each model works best with the specific vocabulary it was trained on, so swapping in a different tokenizer can produce different token ID sequences and unreliable output. When you're choosing an LLM for your app, the tokenizer comes as a package deal.Three methods of tokenizationThe pipeline above handles the mechanics, but the key design decision is step two: how text gets segmented into tokens. There are three approaches, each with different trade-offs.Word-level tokenizationWord-level tokenization treats each unique word as a token. The problem is vocabulary size. In traditional word-level tokenization, any word not in the vocabulary may be mapped to an unknown token. Morphological variations like "run," "running," and "ran" are treated as separate tokens.Character-level tokenizationCharacter-level tokenization makes each character its own token. This handles any input, but sequences get much longer. Since the transformer's attention mechanism scales quadratically with sequence length, this increases compute cost.Subword tokenizationSubword tokenization splits text into units between word-level and character-level. Frequent sequences get their own token, while rarer words decompose into smaller recognizable pieces. This gives you a bounded vocabulary with more efficient handling of morphological variation across languages, which is why subword methods became a practical default for many large language models.The three subword algorithmsThree algorithms implement subword tokenization:Byte Pair Encoding (BPE) iteratively merges the most frequent character pair in the training corpus. It's deterministic and the basis for most production tokenizers. Byte-level BPE starts from byte values rather than Unicode characters, so any character can be represented without producing an unknown token.WordPiece selects merges using a likelihood-based scoring criterion rather than raw frequency alone.Unigram is probabilistic. It scores and prunes vocabulary items based on how well they tokenize training data.These differences matter because they change how text gets split, how large the vocabulary becomes, and how consistently rare words get represented. Vocabulary size also affects efficiency. Models with larger vocabularies tend to produce fewer tokens for the same input.Build fast, accurate AI apps that scaleGet started with Redis for real-time AI context and retrieval