Constraining the output value range of a CNN independent of the loss function

Question

I'm having the following problem: `

I'm training a multi-output CNN and using the relative values of the outputs in my loss function. The net is learning well, but as the absolute values of the outputs are not regularized in anyway in the loss function, the values of the outputs keep rising. This causes a situation where the result I need (values of the outputs relative to each other) are quite right, but the huge absolute values produce underflow resulting NaN values at some point of the training. Is there some way to constrain these absolute values (perhaps elsewhere than the loss function?)

I'm using a custom loss function implementing a recovery angular error metric, so the loss function is somewhat complex. A weighed mean of the net outputs is fed to this function, so using both the net outputs and the weighed mean would lead to some quite problematic gradient derivation. Would it be maybe possible to constrain the value range of the net outputs in the net structure itself?

Answer 1

You can limit the absolute values of the outputs by "punishing" large values in the loss function. This is done by adding an extra term to the loss function.

For example, if your existing loss function is L(yhat, y) where yhat is the output and y the correct value, create a new loss function

L'(yhat, y) = L(yhat,y) + max(0, ||yhat||₁ - k)

When the outputs are "small" the L₁ norm of your output yhat is less than k and imposes no extra loss (keeping everything as before). However once the norm grows above k it starts to impose a loss that guides the training to reduce the absolute values of the outputs values.

You can also limit the values in the network itself by selecting a bounded activation function for the output, for example tanh or sigmoid. However, you need to pick an activation function that suits your problem.