# Training a neural network simultaneously with two different loss functions rather than considering the weighted sum

This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0?

I want to train a neural network that works as an approximator for a function that satisfies a few constraints. For example, I want the network $$\mathcal{N}(x)$$ to satisfy $$\mathcal{N}(x) \leq \eta, \ \forall x \in D_1$$, and $$\mathcal{N}(x) \geq \eta, \ \forall x \in D_2$$, where $$D_1 \cup D_2$$ is the training data set.

So I define a custom loss function as:

$$L_1 := \cup_{x \in D_1} \mathrm{ReLU}( \mathcal{N}(x) - \eta) + \cup_{x \in D_2} \mathrm{ReLU}(-\mathcal{N}(x) + \eta)$$

such that I can generate an output that minimizes the loss and let the training converge when $$L_1=0$$ so that all the conditions are satisfied on the training set (Yes I need this, even at the cost of overfitting?).

Now, I also want the trained neural network to have some Lipschitz bounds, so I consider the log-det barrier method used in the following paper:

https://arxiv.org/abs/2201.00632

Correspondingly, I add an extra loss term $$L_2$$ and try to minimize $$L=\lambda_1L_1+\lambda_2 L_2$$.

However, my question is that if I do this, I may not able to achieve $$L_1=0$$ anymore since $$L_2$$ can be highly negative even if $$L_1$$ is positive can result in minimization of the overall loss function. Is there any way that I can simultaneously train the neural network for $$L_1$$ and $$L_2$$ minimization without using the weighted sum? Or to recover from the final training loss $$L_1$$ and separately use this value for convergence?

Any help in this area is appreciated, I'm not so well-versed with neural networks. Another concern for me is if enforcing $$L_1=0$$ is impossible since $$L_2$$ is kind of like a regularization term that will prevent NN from overfitting, and $$L_1$$ and $$L_2$$ will work against each other.