While researching why we need non linear activation functions, all the explanations revolve around neural network being able to separate values that aren't linearly separable. So I wonder, if we have a neural network whose task is something else, say predicting an output value of a time series, is it still important to have an activation function that is non linear?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
Yes, definitely.
In the simplest example, predicting an output value for a time series is classification. You take in the previous time steps and classify what is the most likely next value. You could do this with a RNN (Recurrent Neural Network) for example.
If the activation functions are all linear, the nerual network is just a glorified linear regression. Think of it like this: a neural network is trying to approximate a complicated function in $n$ dimensional space. It does this by combining operations on a series of known functions, to get a resultant function that hopefully mimics the desired function. The issue with combining linear functions is the only thing you'll ever get at the end is a linear function.
As a concrete example, try and approximate the function $y = x^3 + x^2 -x -1$ by adding a series of linear functions together. You'll find pretty quickly this is useless. However, if you use a non-linear function, such as a ReLU (Rectified Linear Unit) you can quite easily approximate this function. See this implementation on desmos.
If a problem has any sort of complexity to it, the function it follow is likely incredibly complicated, and futile to approximate using linear equations.
-
$\begingroup$ Thanks for you answer. I would just like to check if I'm getting this right, would the desmos implementation represent a neural network with 1 input (value of x), 6 hidden neurons with ReLU activation and 1 output neuron (predicted value of the funciton), where the output of the output neuron is just the sum of hidden neuron outputs? $\endgroup$ Sep 24, 2020 at 4:24
-
$\begingroup$ Yep that's exactly it. And you can imagine the values inside the functions as weights and bias'. For example, for $n_1$ the weight would be -5 and the bias -7.7. Then from the hidden layer to the final layer these weights are either -1 or +1 as seen by the mix of positive and negative in the calculation of $o(x)$ $\endgroup$ Sep 24, 2020 at 5:58
-
$\begingroup$ I don't know if this belongs to a comment, but would it be possible to show the importance of the bias neuron in a similar fashion (on desmos)? $\endgroup$ Sep 24, 2020 at 7:29
-
$\begingroup$ @user1477107 To see the importance, try matching $y=x^2$ using ReLU, but don't use a bias (as in, you can only multiply x by a value, but can never add to it). You'll notice pretty quick that the issue here is that you are stuck with a max(n, x) function, though perhaps scaled differently. You can never reach any sort of complexity. If you try again, but you are able to add a bias, you can align and move around the functions so that they actually match $y=x^2$. Essentially, a bias lets you translate the function $\endgroup$ Sep 25, 2020 at 3:06