Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Most fine-tuning guides answer "how many examples" and skip "how long should each one be." That second question is the one that quietly decides whether your fine-tune helps at inference or fights it. Example length isn't a property you inherit from your data — it's a design choice, and the default (whatever length your dumped transcripts happen to be) is usually wrong in one of two expensive directions.
The framing I keep coming back to: the context window is a cache, not a memory. Fine-tuning changes what the weights know; it does not change the fact that at inference the model reasons over whatever you put in the window right now. Size your training examples to the window you'll actually serve, or you're training for a world you won't deploy into.
The two failure directions
Too short is the sneakier one. Say your real requests arrive with 6–8K tokens of retrieved context, but your training examples are tidy 800-token snippets because that's what your labeling tool exported. You've now fine-tuned a model whose learned prior is "the answer is near the top of a short prompt." At inference you hand it 8K tokens and the relevant fact sits at position 5,000, and the model underweights it — not because the base model can't attend that far, but because your fine-tune taught a length distribution that never occurs in production. You optimized the model onto a distribution you will never sample from.
