While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable. So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?


Yes, definitely.

In the simplest example, predicting an output value for a time series is classification. You take in the previous time steps and classify what is the most likely next value. You could do this with a RNN (Recurrent Neural Network) for example.

If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.

As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a ReLU (Rectified Linear Unit) you can quite easily approximate this function. See this implementation on desmos.

If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.

  • $\begingroup$ Thanks for you answer. I would just like to check if I'm getting this right, would the desmos implementation represent a neural network with 1 input (value of x), 6 hidden neurons with ReLU activation and 1 output neuron (predicted value of the funciton), where the output of the output neuron is just the sum of hidden neuron outputs? $\endgroup$ Sep 24 '20 at 4:24
  • $\begingroup$ Yep that's exactly it. And you can imagine the values inside the functions as weights and bias'. For example, for $n_1$ the weight would be -5 and the bias -7.7. Then from the hidden layer to the final layer these weights are either -1 or +1 as seen by the mix of positive and negative in the calculation of $o(x)$ $\endgroup$
    – Recessive
    Sep 24 '20 at 5:58
  • $\begingroup$ I don't know if this belongs to a comment, but would it be possible to show the importance of the bias neuron in a similar fashion (on desmos)? $\endgroup$ Sep 24 '20 at 7:29
  • $\begingroup$ @user1477107 To see the importance, try matching $y=x^2$ using ReLU, but don't use a bias (as in, you can only multiply x by a value, but can never add to it). You'll notice pretty quick that the issue here is that you are stuck with a max(n, x) function, though perhaps scaled differently. You can never reach any sort of complexity. If you try again, but you are able to add a bias, you can align and move around the functions so that they actually match $y=x^2$. Essentially, a bias lets you translate the function $\endgroup$
    – Recessive
    Sep 25 '20 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.