I recently read two papers:



I am confused about the terms Mean Teacher in BYOL and Knowledge Distillation in DINO.

Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?


1 Answer 1


Knowledge Distillation refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the action of using a teacher model to teach the student.

Mean Teacher, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is one methodology of defining the teacher in self supervised knowledge distillation.

  • $\begingroup$ Thank you for your comment ... So in MT, there is no distilled knowledge (e.g : transfer set of soft labels) from the teacher to the student because the teacher also needs to learn? And both teacher and student are trained together via parameters update (from loss backprogation for the student and EMA for the teacher) $\endgroup$ Jan 6, 2022 at 16:06
  • $\begingroup$ @ĐặngHuyHoàng correct $\endgroup$
    – DKDK
    Jan 6, 2022 at 16:06
  • $\begingroup$ @ĐặngHuyHoàng this is an answer, and not a comment; please see What should I do when someone answers my question? $\endgroup$
    – desertnaut
    Jan 18, 2022 at 15:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .