"How Text Becomes Numbers: Tokenization Explained"

This article covers how raw text becomes something that an NLP model can actually compute with.

Why Computers Cannot Understand Words?

A neural network, at its core, is a system of matrix multiplications and other numerical operations. It has no native concept of a "word" — it only operates on numbers. Feed it the string "cat" directly and a neural network will not be able to multiply a matrix by the letters c-a-t.

This means every NLP system, regardless of whether it is Rule-based, Statistical (n-gram), RNN / LSTM or Transformer — needs a conversion step: raw text in, numbers out. Tokenization is that conversion step. It is the process of splitting text into discrete units (tokens) and assigning each one a unique numerical ID.

Vocabulary