I found a rather odd piece of code in a 3.8k star repo of the well known StyleGAN 2 paper.
In the loss function they use the following expression:
with torch.autograd.profiler.record_function(name + '_backward'):
(real_logits * 0 + loss_Dreal + loss_Dr1).mean().mul(gain).backward()
(They use something similar in line 92)
Multiplying real_logits with 0 seems to me like unnecessary computation overhead. However, the same code is adapted by other researches like the authors of "MAT: Mask-Aware Transformer for Large Hole Image Inpainting". See here:
with torch.autograd.profiler.record_function(name + '_backward'):
((real_logits + real_logits_stg1) * 0 + loss_Dreal + loss_Dreal_stg1 + loss_Dr1 + loss_Dr1_stg1).mean().mul(gain).backward()
The only hypothesis that I have is that they want a quick way of initializing a matrix of a specific shape filled with zeros. However, this makes no sense as everything ends up in the mean() anyways.
What other purpose does this fulfill?
.backwards()is probably not callable, if you created the tensor from zeros. multiplying with zero maybe keeps that intact? $\endgroup$