I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not? The non-linear activation is required to scale the output.
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
A non-linear layer is a matrix multiplication followed by an activation function. A linear layer is just a matrix multiplication.
You have a matrix multiplication, then another matrix multiplication, then an activation function.
Two matrix multiplications in series are equivalent to one matrix multiplication, so your network is equivalent to one non-linear layer.
$\begingroup$
$\endgroup$
1
You are speaking about generalized linear models (GLM)
-
$\begingroup$ Do the model GLM follow semi linearity? I am new to this. Could you please help to provide detail on how it works? $\endgroup$ Mar 19 at 12:56
$\begingroup$
$\endgroup$
If you have a sigmoid activation in your output layer, and linearities before, you can interpret it as a logistic regression model for binary classification (if output dimension is one.)
Also, in practice, having multiple linear hidden layers is a waste of resources because you can define an equivalent linear layer whose weight matrix is the linear combination of those.
Moreover, intermediate linear layers can be useful to "adjust" (i.e. expand or decrease) the output dimension of the previous layer (including the input layer.)
In general, you want to have multiple non-linear layers (i.e. whatever layer with non-linear activation on top) in the middle of your model (i.e. as hidden layers) because the activation of the output layer is often (if not always) problem dependent: e.g., in regression it can be linear.
Having multiple non-linear functions in your neural-net allows it to model (learn) more complex functions, so exploiting the universal function approximation thing: indeed, assuming a correct architecture, loss function, and optimization procedure.
