0
$\begingroup$

Assume a teacher model and a student model. Teacher is bigger than student in terms of depth and/or width, however, comes from the same "family".

In addition to the loss that involves the ground-truth labels, I also want to distill intermediate output feature maps. Assume an intermediate feature map of a teacher of shape [teacher_c,h,w] and a feature map of the student [student_c,h,w], what's the best way to distill different number of channels given same feature map resolution?

If channels were the same, we could use L2 or something else, but this is not the case here.

One option can be to avg pool to [h,w], then L2, but that loses a lot of info.

Another option is pair-wise L2, normalized: L2 between pairs of teacher<>student channels (basically teacher_c_i*student_c_j MSE of feature-maps.

Any other suggestions?

$\endgroup$

1 Answer 1

0
$\begingroup$

My student model has half of the sizes of my teacher model. Hence, I would duplicate student_c to match teacher_c and do L2, kind of like the reverse version of avg pool option without losing information.

I doubt the "pair-wise L2" idea to be reasonable. A student channel being close to one teacher channel while far away from another teacher channel is well needed, meaning that the loss should be zero. However, it can have a high loss due to the far teacher channel.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .