enter image description here

I got this slide from CMU's lecture notes. The $x_i$s on the right are inputs and the $w_i$s are weights that get multiplied together then summed up at each hidden layer node. So I'm assuming this is a node in the hidden layer.

What is the mathematical reason for taking the sum of the weights and inputs and inputting that into a sigmoid function? Is there something the sigmoid function provides mathematically or provides some sort of intuition useful for the next layer?


Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e

layer_output = Weights.layer_input + bias

Consider a 2 layer neural network, the outputs from layer one will be: x2 = W1*x1 + b1 Now we pass the same input to the second layer, which will be

x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b

Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.

So what is the solution?

We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:

  • ReLU
  • Sigmoid
  • tanh
  • Softmax

and there are a lot more of them.

Yay! Our network is no more linear!

We have a lot of different non linear functions, and each of them serve a different purpose.

For example, ReLU is simple and computationally cheap. ReLU(x) = max(0, x) Sigmoid outputs are between 0 and 1. tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1 Softmax is usually used if you want to represent any vector as a discrete probability distribution.

Hope you are having a great day!

| improve this answer | |

By itself, I'm not sure it's possible to know. It's possible the slides were old. Or, the intended purpose was to mention how as sigmoid ranges from 0 to 1. Mostly, it looks like it was intended to bring up gradient descent. But it could also be an entry point to the discussion of other methods such as ReLU. Either that or perhaps some sort of norming function.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.