A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why doesn't sigmoid only work when the relation between input and output is "of sigmoid shape"?

  • $\begingroup$ It should, but the chance of data being related by sigmoid is thin. Also, sigmoid function doesn't map data to the an 's' shape- it squeezes data between 0 and 1. $\endgroup$ – Daniel Aug 24 '18 at 23:50

It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input. This is a trivial fact that applies to a single layer perceptron. This is true for for the single layer case for any activation function, also a trivial fact.

When the layer number is between one and infinity (two or more), the theory bifurcates. The identity function becomes a special case: Any number of layers that conform to a first degree polynomial, $ax + b$, can be replaced with a single identity layers. The alternative case, where there are multiple layers that functionally do not conform to a first degree polynomial, $ax + b$, cannot be replaced by a single layer of some equally simple function. The complexity increases geometrically, which is the entire point of multilayer perceptrons.

Under particular constraints, the multilayer perceptron can produce a wide variety of functional behaviors that do not resemble the activation functions of the layers.

For instance, a properly trained network using sigmoid activation functions, with sufficient layer depth and sufficient massive allocation of computing resources, could theoretically approximate the topography of the Himalayas.


Because any number of linear layers can be represented by a single linear layer:

$A_2*(A_1*x +b_1)+B_2 = A_2*A_1*x+ A2*b_1+b_2$

The same is not true if you have non-linear functions.

A linear function is used when the relation between the input and
        output is linear.
            - A sigmoid function can approximate the relationship between the input and output between 0 and 1.
            - Similar to sigmoid , hyperbolic tangent  funtion approximates the relation between -1 and 1.
            - ReLU is a bit different. ReLU makes negative values 0 and keeps the positive values as they are.

View more from here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.