What is the best way to combine or weight multiple losses with gradient descent?

Question

I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them within the sum. Something like: $clip(w_1\nabla_{L_1} + w_2\nabla_{L_2} + w_3\nabla_{L_3}, c)$.

I am thinking of better approaches. My current idea is to clip gradients separately (to avoid having one gradient "overtaking" the others too much), then weigh them, sum them, and finally clip them (with a smaller threshold than the one used for the first clipping). Something like: $clip(w_1clip(\nabla_{L_1},c) + w_2clip(\nabla_{L_2},c) + w_3clip(\nabla_{L_3},c), c_2)$.

I am not sure what the best way to weigh them would be, though. Like having weights $w_i$ proportional to their gradient norm? I'd like to get some suggestions / references.

Answer 1

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy

In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782

A brief summary of the work done on this topic: It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.

I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: https://en.wikipedia.org/wiki/Pareto_front