Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I recently read two papers:
I am confused about the terms Mean Teacher in BYOL and Knowledge Distillation in DINO.
Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?
Knowledge Distillation refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the action of using a teacher model to teach the student.
Mean Teacher, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is one methodology of defining the teacher in self supervised knowledge distillation.
