Let us suppose we have a network without any functions in between. Each layer consists of a linear function. i.e
layer_output = Weights.layer_input + bias
Consider a 2 layer neural network, the outputs from layer one will be:
x2 = W1*x1 + b1
Now we pass the same input to the second layer, which will be
x3 = W2x*2 + b2
Also x2 = W1*x1 + b1
Substituting back, we have:
x3 = W2(W1*x1 + b1) + b2
x3 = (W2W1)*x1 + (W2*b1 + b2)
x3 = W*x1 + b
Oh no! We still got a linear function. No matter how many layers we add, we will still get a linear function. In that case, our network will never be able to approximate any non linear functions.
So what is the solution?
We will simply add some non linear functions in between. These functions are called activation functions. Some of these functions include:
and there are a lot more of them.
Yay! Our network is no more linear!
We have a lot of different non linear functions, and each of them serve a different purpose.
For example, ReLU is simple and computationally cheap.
ReLU(x) = max(0, x)
Sigmoid outputs are between 0 and 1.
tanh is similar to sigmoid, but zero centered, with outputs from -1 to 1
Softmax is usually used if you want to represent any vector as a discrete probability distribution.
Hope you are having a great day!