Production logs are the best training data you will ever have and the most dangerous. They are real inputs in the real distribution — and they are full of your current model’s outputs, which is exactly what you must not train on.
Selection is the entire job
A log dump is not a dataset. Ninety percent of production traffic is the model already doing the right thing, and training on it teaches nothing you do not already have. The examples worth including are the ones carrying information the model currently lacks.
Requests where a human edited the output. The highest-signal source in any product that has a review step. The edit itself is the label: input plus the corrected output is a perfect training pair, and the diff tells you which failure mode you are fixing.
Requests that failed a validator. Schema violations, failed JSON parses, outputs that broke a downstream contract. Pair the input with a hand-written or repaired correct output.
