I recently read two papers:



I am confused about the terms Mean Teacher in BYOL and Knowledge Distillation in DINO.

Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?


Knowledge Distillation refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the action of using a teacher model to teach the student.

Mean Teacher, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is one methodology of defining the teacher in self supervised knowledge distillation.

  • $\begingroup$ Thank you for your comment ... So in MT, there is no distilled knowledge (e.g : transfer set of soft labels) from the teacher to the student because the teacher also needs to learn? And both teacher and student are trained together via parameters update (from loss backprogation for the student and EMA for the teacher) $\endgroup$ Jan 6 at 16:06
  • $\begingroup$ @ĐặngHuyHoàng correct $\endgroup$
    – DKDK
    Jan 6 at 16:06
  • $\begingroup$ @ĐặngHuyHoàng this is an answer, and not a comment; please see What should I do when someone answers my question? $\endgroup$
    – desertnaut

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.