# Types of activation functions used in neural networks

I need to specify the points and conditions that require me to use one activation function over another,

• Why is it not possible to use the sigmoid FUNCTION in hidden layers, and it is preferable to use Relu instead?
• When do we use the Relu function in hidden layers and when do we use Tanh?
• Here is general guide/practice, turing.com/kb/…. Please add the task you want to use your neural network and what model under your consideration in your question. It will help us to give you good suggestion. Nov 30, 2023 at 19:58

Let me give a very easy explanation... Activation Functions in neural networks are useful as they introduce non-linearity into the model, which helps in complex data pattern understanding.

Sigmoid Function

• Sigmoid converts output values into range of (0,1).
• It is mostly used for binary classification problems.
• It suffers from Vanishing Gradient Problem. As the Input value becomes small or large. The gradient turns out very very small (near about zero).This will slow down the training process because weights and biases receives very small updates.
• E.g. : For a large input, say 10, sigmoid(10) ≈ 0.9999. The gradient is sigmoid(10) * (1 - sigmoid(10)) ≈ 0.0001, which is very small.

ReLU Function

• ReLU converts all the negative input to 0 and positive as it is
• It is mostly used in hidden layers which allows faster training because of the linearity in non-saturating form.
• It helps in the vanishing gradient problem of hidden layers.
• In some cases, there is a dyeing ReLU problem, where neurons sometimes only output zero(for all negative inputs).
• E.g. : For an input of 5, ReLU(5) = 5. For an input of -3, ReLU(-3) = 0, and the gradient is 0.

Tanh Function

• The Tanh function outputs values in the range of (-1,1). It is somewhat similar to sigmoid but stretched for a wider range
• It is very useful when data is centered around zero, Useful in situations where input is negative.
• Tanh also suffers from vanishing gradient but lesser extent.
• E.g. :For a large input, say 10, tanh(10) ≈ 1. The gradient is 1 - tanh(10)^2 ≈ 0, which is very small.

Other There are other activation functions like

• Leaky ReLU: It is a variation of ReLU but allows a small negative result. This will prevent dead neuron problem.
• PReLU(Parametric ReLU): It is modified Leaky ReLU where small slope of negative input is learned during training.
• ELU(Exponential Linear Unit): Positive input will be as it is and for negative result exponential decay function is used. It is combination of sigmoid/tanh with ReLU.
• SELU(Scaled Exponential Linear Unit): It will encourage self-normalizing properties for faster and more stable training.
• SoftMax: Converts vector of values into a probability distribution. Where probability is in proportion to its exponential. Mostly used in the output layer of multi-class classification.
• Others are SoftPlus, Swish, Hard Sigmoid, Hard Swish, Gaussian, etc...