In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one.

There are several strategies to do this:

  • Soft distillation

    Given logits of the teacher one adds the KL-divergence between the student logits and teacher logits to the loss: $$ \mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + \lambda \mathcal{L}_{KL} (y_{student}, y_{teacher}) $$ Intuition behind this approach is clear - logits are more informative than a single target label and seemingly allow for faster training.

  • Hard distillation

    One adds the BCE between student logits and teacher model outputs as if they were true labels. $$ \mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + \lambda \mathcal{L}_{BCE} (y_{student}, y_{teacher}) $$

And the benefit of the last approach is unclear to me. For the perfect model, one will have no difference with the vanilla training procedure, and for the case, where the teacher makes mistakes, we will optimize the wrong objective.

Despite these concerns, it was shown experimentally in several papers, and in the Deit, that this objective can improve performance. Even more, it is better, than soft distillation.

enter image description here

Why can this be the case?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.