A few weeks ago my friend @u84u and I (@ronak-create) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text by tokenizing it into subwords instead of raw bytes, why don't we tokenize text before handing it to a byte-level compressor like LZMA or zstd?
It felt like the kind of idea someone must have already tried and quietly dropped. So instead of writing a blog post about the idea, we built the harness to actually test it, and the result is parmar — a subword-tokenization pre-filter for byte-level compressors, plus a fairly paranoid benchmarking rig to check whether the idea holds up at scale.
Short version: it works, but for a narrower reason than we expected, and the harness told us that just as clearly as it told us the headline number.
The hypothesis
Byte-level compressors like LZMA2 and zstd find repeated patterns inside a fixed-size sliding "dictionary window," measured in bytes. Our premise: if you replace UTF-8 prose with BPE token IDs — the same tokenization used to feed LLMs — before compressing, the token stream is roughly 45% smaller than the source text. A 64 MiB dictionary window that normally covers ~64 MB of prose can now cover roughly twice as much prose once that prose has been pre-shrunk.
