How One Bad Chunk Can Triple Your LLM Bill
After Episode 2, I trusted my extraction pipeline. Clean text, tables intact, headers stripped out. I felt like I'd earned the right to stop worrying about the early stages and move on.
So I moved on to chunking, expecting it to be the boring part. Split the text, pick a size, done.
The first version I shipped used a small chunk size on purpose. The reasoning felt sound: smaller chunks mean each one is more focused, retrieval should be more precise, nothing gets buried under irrelevant text. Felt like the safe choice.
It didn't feel small until I ran it across a real batch of documents. A handful of test PDFs turning into a few hundred chunks looks completely fine on a laptop. The same chunk size, run across a production-sized document set, turned into far more chunks than I'd actually sat down and calculated in advance — and the ingestion job I expected to finish in a few minutes was still running an hour later, CPU pinned the whole time, one embedding call after another, after another.
