Every Word I Say Gets Tokenized. This Library Does It 1000x Faster.

GigaToken is a drop-in replacement for HuggingFace Tokenizers and tiktoken that’s up to 1000x faster. I looked inside from an AI perspective.

Raccontata dadev.to