NLP models cannot process words directly, which is why tokenization[1] exists. Tokenization ends with a list of integers — token IDs like [30642, 1634, 318, ...]. That list is what actually gets fed into a model. This article answers why NLP models cannot process raw text directly.
The answer is simple: NLP models consist of neural networks that perform matrix multiplication, which needs a list of numbers, not characters.
One Layer of a Neural Network: Multiply, Add, Then a Small Nonlinear Step
A neural network is built from layers, and each layer does one core operation: take a vector of numbers in, multiply it by a matrix of learned weights, add a bias, and produce a new vector of numbers out.
Example
