I know that if you use an ReLU activation function at a node in the neural network, the output of that node will be non-negative. I am wondering if it is possible to have a negative output in the final layer, provided that you do not use any activation functions in the final layer, and all the activation functions in the previous hidden layers are ReLU?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes, if there's no activation function in the last layer, the weights could simply be negative there, so the network would multiply a positive value with a negative weight, therefore outputting a negative value.
There is still an activation function, but it is the identity.
$\begingroup$
$\endgroup$
I guess you are using NN for Regresions. In the most common aplication a scale of the outputs is implemented. This is recommended. Specialy if you have more than one output with diferent scales. Otherwise, you will remunerate the neural network for correcting the error of one variable over the other. If you still want to avoid a scale of the outputs. Yes. You can use the identity function in the output layer or a linear function (tha same with different slope). The weights and bias of some conections will become negative and the hidden neurons are going to work as always.