0
$\begingroup$

This seems like a silly, trivial question, but I just want to confirm it in case I'm missing something.

I'm trying to train a ReLU neural network, which is supposed to be a function that satisfies some conditions. For example, I want the network $\mathcal{N}(x)$ to satisfy $\mathcal{N}(x) \leq \eta, \ \forall x \in D_1$, and $\mathcal{N}(x) \geq \eta, \ \forall x \in D_2$, where $D_1 \cup D_2$ is the training data set.

I incorporate these conditions as a custom loss function as follows:

$$L := \cup_{x \in D_1} \mathrm{ReLU}( \mathcal{N}(x) - \eta) + \cup_{x \in D_2} \mathrm{ReLU}(-\mathcal{N}(x) + \eta) $$

The idea is that if $L = 0$, then the conditions are satisfied for $\mathcal{N}(x)$. So when I train the network, I ensure that the epoch loss is 0 (that's when the algorithm converges). If it matters, I use an SGD optimization technique for training.

So it is safe to say that after training, the network for sure satisfies those conditions with a 100% guarantee over the training data set, right?

I understand that this may not be the case for random unseen data, but I just want to make sure that the guarantee holds for the training data set, and it will not be required to validate this a posteriori.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Sep 20, 2022 at 12:22
  • $\begingroup$ Interesting loss term you've got there. Regardless, are you asking whether this is the case without any further verification? Otherwise, you can just calculate $L$ after the training on $\mathcal{N}$ was completed. Also, I think that if you're using batches, the loss term is in fact a union over $x\in B$, which may result in some non-zero values over all $D_1$ $\endgroup$
    – Hadar Sharvit
    Commented Sep 20, 2022 at 13:26
  • $\begingroup$ @HadarSharvit Can you explain how it may result in non-zero values? I do in fact use batches. So my data set $D_1$ is divided into let's say batches $B_1$, $B_2$ and $B_3$ (randomly shuffled in each epoch), epoch loss is 0 only if the batch losses $B_1, B_2$ and $B_3$ are 0, so that means it should be 0 over $D_1$. Am I missing something? $\endgroup$
    – Acad
    Commented Sep 20, 2022 at 13:34
  • $\begingroup$ @HadarSharvit and yes I want to avoid further verification! $\endgroup$
    – Acad
    Commented Sep 20, 2022 at 13:36
  • 1
    $\begingroup$ Yes, If the epoch loss is minimized to $0$, then the conditions are satisfied for $N(x)$ & it is safe to say that after training, the network for sure satisfies those conditions with a 100% guarantee over the training data set. However, this does not mean that the network will be 100% accurate on new, unseen data. $\endgroup$
    – Faizy
    Commented Oct 17, 2022 at 11:20

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .