This is a follow up on the already asked question: Is the neural network 100% accurate on training data if epoch loss is minimized to 0?

I want to train a neural network that works as an approximator for a function that satisfies a few constraints. For example, I want the network $\mathcal{N}(x)$ to satisfy $\mathcal{N}(x) \leq \eta, \ \forall x \in D_1$, and $\mathcal{N}(x) \geq \eta, \ \forall x \in D_2$, where $D_1 \cup D_2$ is the training data set.

So I define a custom loss function as:

$$L_1 := \cup_{x \in D_1} \mathrm{ReLU}( \mathcal{N}(x) - \eta) + \cup_{x \in D_2} \mathrm{ReLU}(-\mathcal{N}(x) + \eta) $$

such that I can generate an output that minimizes the loss and let the training converge when $L_1=0$ so that all the conditions are satisfied on the training set (Yes I need this, even at the cost of overfitting?).

Now, I also want the trained neural network to have some Lipschitz bounds, so I consider the log-det barrier method used in the following paper:


Correspondingly, I add an extra loss term $L_2$ and try to minimize $L=\lambda_1L_1+\lambda_2 L_2$.

However, my question is that if I do this, I may not able to achieve $L_1=0$ anymore since $L_2$ can be highly negative even if $L_1$ is positive can result in minimization of the overall loss function. Is there any way that I can simultaneously train the neural network for $L_1$ and $L_2$ minimization without using the weighted sum? Or to recover from the final training loss $L_1$ and separately use this value for convergence?

Any help in this area is appreciated, I'm not so well-versed with neural networks. Another concern for me is if enforcing $L_1=0$ is impossible since $L_2$ is kind of like a regularization term that will prevent NN from overfitting, and $L_1$ and $L_2$ will work against each other.



You must log in to answer this question.

Browse other questions tagged .