Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is a node in the hidden layer.
What is the mathematical reason for taking the sum of the weights and inputs and inputting that into a sigmoid function? Is there something the sigmoid function provides mathematically or provides some sort of intuition useful for the next layer?
Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
layer_output = Weights.layer_input + bias
Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be
x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b
Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.
So what is the solution?
We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:
and there are a lot more of them.
Yay! Our network is no more linear!
We have a lot of different non linear functions, and each of them serve a different purpose.
For example, ReLU is simple and computationally cheap. ReLU(x) = max(0, x) Sigmoid outputs are between 0 and 1. tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1 Softmax is usually used if you want to represent any vector as a discrete probability distribution.
Hope you are having a great day!
By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming function.
