The Sequence Knowledge - 928: The Missing 5%: Why Distillation Is Harder Than It Looks

Distillation can make models smaller, faster, and cheaper. The difficult part is deciding what the student cannot afford to forget.

Raccontata dathesequence.substack.com