A model release arrives with an irresistible claim: a 7-billion-parameter student retains 95 percent of the performance of a 70-billion-parameter teacher.This sounds like one of the best trades in computing. Ten times smaller. Nearly as intelligent. Put it on a laptop, inside an agent loop, or behind an API with much better margins.But where did the missing 5 percent go?Perhaps the student keeps the teacher’s mathematics score but loses its ability to know when it is confused. Perhaps it solves familiar coding tasks but cannot recover after a failed tool call. Perhaps it imitates safety language without inheriting the judgment behind it. Perhaps it produces beautiful reasoning traces that collapse when the problem takes an unfamiliar turn.The missing 5 percent may not be distributed evenly. It may contain the bridge, not the road.
The Sequence Knowledge - 928: The Missing 5%: Why Distillation Is Harder Than It Looks
Distillation can make models smaller, faster, and cheaper. The difficult part is deciding what the student cannot afford to forget.
133 words~1 min read
