Questions tagged [vanishing-gradient-problem]

For questions related to the vanishing gradient problem, which is a numerical problem that occurs while training a (deep) neural network with a gradient-based optimization technique. There's also the related exploding gradient problem.

7 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
41 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
3
votes
0answers
43 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
2
votes
0answers
63 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
2
votes
1answer
50 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
1
vote
0answers
26 views

How do I infer exploding or vanishing gradients in Keras?

It may already be obvious that I am just a practitioner and just a beginner to Deep Learning. I am still figuring out lots of "WHY"s and "HOW"s of DL. So, for example, if I train a ...
1
vote
0answers
94 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
0
votes
0answers
30 views

In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?

There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs. Both of these sources mention the reason LSTMs are ...