I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is so in my case. How does backpropagation work in this case? For a small change in weights, the change of the l1 component would obviously always be far greater than the SSIM component. So, it seems that only l1 part will affect the learning and the SSIM part would almost have no role to play. Is this correct? Or I am missing something here. I think I am, because in the DSSIM implementation of Keras-contrib, it is mentioned that we should add a regularization term like a l2 loss in addition to DSSIM (https://github.com/keras-team/keras-contrib/blob/master/keras_contrib/losses/dssim.py); but I am unable to understand how it would work and how the SSIM would affect the backpropagation being totally overshadowed by the large magnitude of the other component. It will be a great help if someone can explain this. Thanks.
1 Answer
Don't know if you have this doubt anymore, but this would be helpful for those who are facing similar problems-
You will need to find the correct weights with which you add these two loses by hyperparameter search. That is, find the best $\lambda$ for the loss-
$$ L = Loss_1 + \lambda(Loss_2) $$
Here $Loss_1$ and $Loss_2$ can be any losses. Here, we take them as SSIM and L1-regularization losses, respectively. You can keep the gradient flow of the regularization loss below some percentage of that of $Loss_1$ by choosing the correct value of gradient clipping for a hyperparameter. Note that by setting this hyperparameter to low, you may even impede its performance(the exact opposite of the case mentioned here). For this specific case, L1 regularization has a constant gradient, equal to the hyperparameter itself. So, by keeping it around 10% of the max gradient(or 10% of the max loss generally works as well), we should not face these kinds of problem.
