As in sigmoid function when input x is very large or very small the curve is flat that means low gradient descent but when it is in between the slope is more so, My question is how this thing helps us in neural network.


There are several nice things about using the logistic function and a few drawbacks as well. I'm not going to discuss all of them, but here is a small synopsis.

Nice Properties

enter image description here

For networks which calculate probabilities, it brings the input features into a nice 0 to 1 range.

Since the logistic function maps inputs to the range (0,1), it is commonly used as an activation function in neural networks which require some sort of classification. For example, if we had a convolution neural network (CNN) that classified if an image given to it was a dog or not, it could have an output neuron that returned 0 if the image was not a dog or 1 if it was. When the CNN is given an image with inputs of varying dimensions and ranges, it's more intuitive to deal with the input features in the realm of 0 to 1 instead of 0 to 255 or whatever input format you're dealing with.

It is smooth and differentiable.

When performing back propagation, it's much easier to deal with differential functions. Otherwise you'd have to approximate the gradient which may lead to unfavorable results.

A Vanishing Issue

At extreme values, the gradient vanishes.

The OP was also hinting that at very positive and very negative values of X, derivative of the logistic function could become close to 0. For large input values this may hinder learning for extreme values. If you think of gradient decent, the jump that gradient descent makes is only as strong as its magnitude of the gradient that it requires. Analytically speaking, this also hurts back propagation because learning is calculated using the chain rule, which ends up multiplying several gradients together.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.