# Multi-objective training involving maximization of one loss function and minimization of another

I need my model to predict $$s$$ from my data $$x$$. Additionally, I need the model to not use signals in $$x$$ that are predictive of a separate target $$a$$. My approach is to transform $$x$$ into a representation $$\Psi(x)$$ such that it's good at predicting $$s$$ but bad at predicting $$a$$.

Concretely, let the prediction for $$s$$ be $$$$\hat{s} = (\Gamma \circ \Psi)(x),$$$$ and that for $$a$$ be $$\hat{a} = (\Omega \circ \Psi)(x),$$ where $$\Psi, \Gamma, \Omega$$ are implemented as MLP layers. Further, let $$\ell_s(\Psi,\Gamma)$$ and $$\ell_a(\Psi,\Omega)$$ be the loss functions corresponding to $$s$$ and $$a$$, respectively.

The objective thus is to simultaneously

• minimize $$\ell_s(\Psi,\Gamma)$$ (good at predicting $$s$$), and
• maximize $$\min_\Omega \ell_a(\Psi,\Omega) := -f(\Psi)$$ (bad at predicting $$a$$)

There may not exist a $$(\Psi, \Gamma)$$ combination that simultaneously achieves both. So the hope is to find something on the Pareto frontier by linearly combining the objectives: $$$$\tag{*}\label{*} \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) + \alpha f(\Psi) \right] = \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) - \alpha \min_\Omega \ell_a(\Psi,\Omega) \right]$$$$ where $$\alpha > 0$$ is a hyperparameter that trades off the two original objectives. However, I'm not sure how to train the layers $$\Psi, \Gamma, \Omega$$ to achieve \eqref{*}. One procedure I could think of is to alternate between

1. Freeze $$\Psi, \Gamma$$ and tune $$\Omega$$ to minimize $$\ell_a(\Psi,\Omega)$$,
2. Freeze $$\Omega$$ and tune $$\Psi, \Gamma$$ to minimize $$\ell_s(\Psi,\Gamma) - \alpha \ell_a(\Psi,\Omega)$$

until convergence. Would that work?

It's probably important that both steps be able to tune the learned representation Ψ. Also, that you are always minimizing the combined loss.

You should do:

1. Freeze Γ and tune Ψ,Ω to minimize ℓ𝑠(Ψ,Γ)−𝛼ℓ𝑎(Ψ,Ω),
2. Freeze Ω and tune Ψ,Γ to minimize ℓ𝑠(Ψ,Γ)−𝛼ℓ𝑎(Ψ,Ω).

Or, better yet, if Γ and Ω accept the same examples, to just train both in each training step.

Your setup is fine, and sensible. For example, in learning voice conversion, some architectures will have a module adds the following to the objective: This representation of the speech should make it difficult to predict who the speaker is (and thus only model the semantic content of the speech, not speaker style).

These Greek caps seem like a strange notation to me, btw.

• $\Gamma$ and $\Omega$ does accept the same examples. However, if I train all the layers with $\ell_s - \alpha \ell_a$ as the objective to be minimized, I'm not sure it'll achieve \eqref{*}. I need the $\Psi$ layer to be such that $\Omega$ cannot be tuned further to lower $\ell_a$. This is what ensures that $\Psi$ has been optimized to prevent prediction of $a$. Aug 29, 2022 at 4:04
• Again, your goal is to induce a representation Ψ that is maximally effective at predicting s using Γ and maximally confusing at predicting 𝑎 using Ω, weighted by 𝛼. It doesn't really matter if you train back and forth using freezing, or jointly without freezing. Training jointly without freezing is more elegant. I can try to find examples from the VC literature showing this is fine, but you haven't presented an argument why it shouldn't work. Rather than hypothesizing endlessly, why not just try both approaches and see how they work? If you can't evaluate these approaches, let's discuss that Aug 30, 2022 at 1:46
• Here is the simple reason it won't work: $$\min_{\Psi,\Gamma, \Omega} \left[ \ell_s(\Psi,\Gamma) - \alpha \ell_a(\Psi,\Omega) \right] \neq \min_{\Psi, \Gamma} \left[ \ell_s(\Psi,\Gamma) - \alpha \min_\Omega \ell_a(\Psi,\Omega) \right]$$ The LHS is a minima while the RHS \eqref{*} is a saddle point. If you try gradient descent with this objective, the $\Omega$ layer gets updated as $\Omega \leftarrow \Omega \color{red}{+} \eta \; \alpha \nabla_\Omega\ell_a$. Intuitively, you want $\Omega$ to be the best it can possibly be (for the chosen $\Psi$) at predicting $s$ and still fail miserably. Aug 30, 2022 at 4:46
• Training all layers jointly is my preference too. In fact, my first attempt was just that with $\ell_s -\alpha \ell_a$ as the objective. It's when I couldn't get it to converge that I realized it's not even what I need. Now, there is this [DANN paper][1] someone just directed me to with a neat gradient reversal layer trick that I think should work for my case too. I haven't quite got it working yet though. [1]: arxiv.org/abs/1505.07818 Aug 30, 2022 at 4:48