Take the number 1234567. Now take 234567. To you, the second is the first with the leading digit removed — the trailing six digits are identical, and their place values are identical.
To a GPT-lineage tokenizer, they share almost nothing. The first splits as 123|456|7. The second splits as 234|567. Not one token boundary lines up. The model has to relearn "these are the same trailing digits" from scratch, per length class.
That is digit tokenization, and it is the single most under-discussed reason LLMs fumble arithmetic on numbers longer than three digits.
TL;DR
Digit tokenization groups digit runs into chunks (up to 3 for tiktoken-lineage tokenizers) using a left-to-right greedy pre-tokenizer regex. Chunk boundaries therefore depend on total number length, not on place value.
