Skip to main content
added 726 characters in body
Source Link

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy

In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782

A brief summary of the work done on this topic: It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.

I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: https://en.wikipedia.org/wiki/Pareto_front

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy

In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy

In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782

A brief summary of the work done on this topic: It is often better to analytically combine the gradients, than to directly combine the losses and then get one gradient. Many ways have been proposed to do this, some involving gradient norms, some involving dynamic rescaling of gradients, some involving geometric analysis, and some involving game theory.

I should also note: there is an entirely different approach to multi objective optimization called Pareto Optimality, where instead of simply combining the losses in one particular way, we analyze a set of different ways of combining the objectives and their trade-offs. For more info, this is a good starting point: https://en.wikipedia.org/wiki/Pareto_front

Source Link

This is an important subfield within multi-task learning, called gradient combination. Here is a list of about a dozen recent approaches: https://github.com/Manchery/awesome-multi-task-learning#loss--gradient-strategy

In particular, this paper is a good starting place: https://arxiv.org/abs/2001.06782