5
$\begingroup$

I am training a deep neural network. There is a constraint on an output value of the network. (e.g. Output has to be between 0 and 180) I think some possible solutions are using sigmoid,tanh activation at the end of the layer. I wonder if there are better ways to put constraints on the output value of a neural network.

$\endgroup$
  • $\begingroup$ Please give more detail about how the output value is interpreted. This will help to understand what kind of loss function and activation of the last layer are suitable. Do you have 181 classes ? Is it a position or an angle ? $\endgroup$ – Andre Holzner Oct 26 '18 at 7:43
2
$\begingroup$

There are many ways of constraining the network's output.

Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.

However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.

You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This can lead to problems, depending on your problem.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.