From the lecture in machine learning I know, that a linear activation function can only produce a linear function, but I don't know if it can produce a connected linear function like the one in the image? This function consists of multiple concatenated lines.

enter image description here

  • $\begingroup$ I don't think so... that would make the output non-linear $\endgroup$ – Thomas W Aug 3 '17 at 9:21

A linear activation would not be able to separate the data like you have shown and more over, it doesn't even matter how many layers you throw into the network.

If we had multiple linearly activated layers, each feeding into each other, the neurons in the previous layer would calculate some weighted sum of the input and send it to the next layer as input where the next layer's neurons would also calculate a weighted sum on that input and it in turn, fires based on another linear activation function.

No matter how many layers and neurons there are, if all are linear in nature, the final activation function of last layer is also a linear function of the input of first layer.

That means that any or all of these layers can be replaced by a one layer. This completely looses the advantage of stacking layers because any multilayer network is equivalent to a single layer with linear activation because a combination of linear functions in a linear manner is still another linear function.

A good way to visualize this messing around with Tensorflow playground, which has a spiral data set similar to your data.

In contrast to the failure of linear activation functions for this data in the previous link, check out this much smaller network using a Tanh activation function which can separate the data within 100 or so iterations.

For further reference on activation functions, checkout this blog post.

  • $\begingroup$ how does this answer fit with the Universal Approximation Theorem? The OP's space in bounded, and the curve is continuous, so a single-hidden-layer network of sufficient width should be capable of learning the curve even with linear activation. $\endgroup$ – beldaz Apr 18 at 21:48

The question title is:

Can a NN with linear activation functions produce a connection of linear functions?

The answer is YES IT CAN. With multi-layer NNs, you can approximate nonlinear functions such as the one in the figure, even with linear activations.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.