On-policy distillation has become one of the standard recipes for training small reasoning models. If outcome-level reinforcement learning with verifiable rewards gives you sparse feedback only at the end of a long chain of thought, on-policy distillation offers dense token-level advantages. The student generates rollouts, and a larger teacher model scores every token along the path.
A new paper from Purdue researchers Yi Ding and Ruqi Zhang examined what that teacher actually does during training.
Their finding is that teacher supervision in on-policy distillation is largely noise, and larger teacher models produce more noise, not less. Even more surprising: when they stripped out the teacher entirely and replaced its advantage scores with a simple negative penalty on low-probability tokens, the student achieved identical or better performance.
What the Teacher Actually Emits
In standard on-policy distillation, the student model samples a full reasoning trajectory. The teacher then evaluates the sequence and assigns an advantage score to each token to guide the policy gradient.
