The idea in plain language.
A teacher's probability distribution contains more information than its single winning answer.
How it works
A large model or ensemble predicts a distribution over possible answers. Instead of teaching a student only the correct class, distillation also trains it against the teacher's softened predictions. Relative probabilities can communicate which alternatives the teacher considers similar. The student aims to retain useful behavior while requiring less computation at deployment.
What to keep in mind
The student inherits limitations of its teacher and training data. Successful compression on one task does not establish equal performance everywhere.
Source: Distilling the Knowledge in a Neural Network. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.