4
$\begingroup$

I am training a deep neural network. There is a constraint on the output value of the neural network (e.g. the output has to be between 0 and 180). I think some possible solutions are using sigmoid, tanh activation at the end of the layer.

Are there better ways to put constraints on the output value of a neural network?

Improve this question
$\endgroup$
1
  • $\begingroup$ Please give more detail about how the output value is interpreted. This will help to understand what kind of loss function and activation of the last layer are suitable. Do you have 181 classes ? Is it a position or an angle ? $\endgroup$
    – Andre Holzner
    Oct 26, 2018 at 7:43

1 Answer 1

Reset to default
3
$\begingroup$

There are many ways of constraining the network's output.

Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.

However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.

You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This can lead to problems, depending on your problem.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .