The dangerous fine-tuning failures are not the ones that raise an exception. They are the ones where the run completes, the loss curve looks plausible and the model is quietly wrong — and the way to catch those is to know what each failure mode looks like from the outside.
Failures that crash
Out of memory partway through, not at the start
A run that survives a thousand steps and then raises CUDA out of memory did not gradually leak. It met the longest sequence in the dataset. Memory for activations and for the logits tensor scales with sequence length, and the tail of a real length distribution is long.
Diagnostic: plot your token-length histogram and check whether the crash step corresponds to a batch containing the maximum. Sort the dataset by length and run the longest batch first — if it fails immediately, that was it. Fix: cap max_seq_length, use length-grouped batching, reduce the micro-batch and raise gradient accumulation, or use a chunked cross-entropy that never materialises the full fp32 logits. The arithmetic is in the QLoRA memory budget.
