Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
It is known that the primary purpose of activation functions, used in neural networks, is to introduce non-linearity.
Then how can the linear activation function, especially the identity function, be treated as an activation function?
Are there any special applications/advantages in using an identity function as I cannot see any such use theoretically?
The identity function can be useful in some cases. For example, if you are doing regression, the output of your neural network needs to be a real (or floating-point) number, so you use the identity function. (If you were doing logistic regression or classification, that wouldn't probably be the case). The identity function is also used in the residual networks (see figure 1). There are probably other examples of its usage and usefulness.
Some people may not consider the identity an activation function because it does nothing to the input. However, whether you consider the identity an activation function or not is a matter of convention, in the same way that considering a model with only identity (aka linear) functions a real neural network or a perceptron a neural network is a matter of convention.
I don't think there's still a consensus on this subject. In fact, you will often hear (or see) people say (or write) that they use "no activation (function)" or "linear activation" rather than saying that they use the "identity" (for example, see the documentation for the parameter activation here).
