1
$\begingroup$

I have a simple question about the choice of activation function for the output layer in feed-forward neural networks.

I have seen several codes, where the choice of activation function for the output layer is linear. Now, it might well be that I am wrong about this, but isn't that simply equivalent to a rescaling of the weights connecting the last hidden layer to the output layer? And following this point, aren't you just as well off with just using the identity function as your output activation function?

$\endgroup$
  • $\begingroup$ Do you mean using a linear activation function that is not the identity function? Like f(x)=2x? $\endgroup$ – Mr. Eivind Sep 26 '18 at 15:47
3
$\begingroup$

And following this point, aren't you just as well off with just using the identity function as your output activation function?

When someone declares that the output of a neural network layer is linear this is exactly what they mean. It can also be described as "no activation function".

Saying that a NN layer has linear activation is a kind of short-hand for saying "the whole layer is a linear function of its inputs", which is true without adding any activation function, just using the weights and bias.

There is usually no separate linear function applied, and libraries such as Keras include the term 'linear' only for completeness, or so that the choice can be made explicit in the code, as opposed to an unseen default.

Note that the link to Keras activation definition above says:

Linear (i.e. identity) activation function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.