1
$\begingroup$

I have a simple question about the choice of activation function for the output layer in feed-forward neural networks.

I have seen several codes where the choice of the activation function for the output layer is linear.

Now, it might well be that I am wrong about this, but isn't that simply equivalent to a rescaling of the weights connecting the last hidden layer to the output layer? And following this point, aren't you just as well off with just using the identity function as your output activation function?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Do you mean using a linear activation function that is not the identity function? Like f(x)=2x? $\endgroup$
    – Mr. Eivind
    Sep 26, 2018 at 15:47

1 Answer 1

Reset to default
3
$\begingroup$

And following this point, aren't you just as well off with just using the identity function as your output activation function?

When someone declares that the output of a neural network layer is linear this is exactly what they mean. It can also be described as "no activation function".

Saying that a NN layer has linear activation is a kind of short-hand for saying "the whole layer is a linear function of its inputs", which is true without adding any activation function, just using the weights and bias.

There is usually no separate linear function applied, and libraries such as Keras include the term 'linear' only for completeness, or so that the choice can be made explicit in the code, as opposed to an unseen default.

Note that the link to Keras activation definition above says:

Linear (i.e. identity) activation function.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .