# What is the purpose of hard distillation?

In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one.

There are several strategies to do this:

• Soft distillation

Given logits of the teacher one adds the KL-divergence between the student logits and teacher logits to the loss: $$\mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + \lambda \mathcal{L}_{KL} (y_{student}, y_{teacher})$$ Intuition behind this approach is clear - logits are more informative than a single target label and seemingly allow for faster training.

• Hard distillation

One adds the BCE between student logits and teacher model outputs as if they were true labels. $$\mathcal{L}_{loss} = (1 - \alpha) \mathcal{L}_{BCE} (y_{student}, y_{true}) + \lambda \mathcal{L}_{BCE} (y_{student}, y_{teacher})$$

And the benefit of the last approach is unclear to me. For the perfect model, one will have no difference with the vanilla training procedure, and for the case, where the teacher makes mistakes, we will optimize the wrong objective.

Despite these concerns, it was shown experimentally in several papers, and in the Deit, that this objective can improve performance. Even more, it is better, than soft distillation.

Why can this be the case?