Questions tagged [vanishing-gradient-problem]
For questions related to the vanishing gradient problem, which is a numerical problem that occurs while training a (deep) neural network with a gradient-based optimization technique. There's also the related exploding gradient problem.
21
questions
0
votes
1
answer
32
views
How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?
I am training several different GAN architectures, and I noticed that larger batch sizes may lead to vanishing or exploding gradients. In the interest of accelerating training, however, larger batch ...
0
votes
0
answers
56
views
Are ResNets necessary if we use Batch Normalisation?
One of the issues of very deep neural networks is vanishing gradients. This problem was addressed through ResNet by adding skip connections. However, is this actually necessary if we use batch ...
0
votes
1
answer
43
views
How does not learning far inputs make the RNN forget far inputs?
I am totally aware of the problem of the vanishing gradient. It usually occurs with vanilla RNN, where with a long sequence of data, the gradient will vanish or explode for far input sequence, and ...
- 103
3
votes
1
answer
77
views
Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
While looking into the vanishing gradient problem, I came across a paper (https://ieeexplore.ieee.org/abstract/document/9336631) that used artificial derivatives in lieu of the real derivatives. For a ...
0
votes
0
answers
71
views
Why is the vanishing gradient problem more discussed than the exploding gradient problem in the context of deep CNNs?
Both vanishing and exploding gradient problems may occur if we increase the number of layers in CNN. It is due to the product operation in the chain rule.
Whenever I read recent research papers, i.e. ...
- 3,501
1
vote
1
answer
158
views
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.
The chain rule of ...
- 3,501
0
votes
0
answers
35
views
Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?
There are instances in literature where we need to change loss function in order to escape from gradient problems.
Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
- 3,501
4
votes
0
answers
126
views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
- 183
1
vote
0
answers
136
views
How do I infer exploding or vanishing gradients in Keras?
It may already be obvious that I am just a practitioner and just a beginner to Deep Learning. I am still figuring out lots of "WHY"s and "HOW"s of DL.
So, for example, if I train a ...
2
votes
1
answer
145
views
How does vanish gradient restrict RNN to not work for long range dependencies?
I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like:
Consider ...
- 41
2
votes
0
answers
226
views
How to decide if gradients are vanishing?
I am trying to debug a convolutional neural network. I am seeing gradients close to zero.
How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
- 121
0
votes
0
answers
61
views
In LSTMs, how does the additive property enables better balancing of gradient values during backpropagation?
There are two sources that I'm using to to try and understand why LSTMs reduce the likelihood of the vanishing gradient problem associated with RNNs.
Both of these sources mention the reason LSTMs are ...
- 167
3
votes
0
answers
74
views
Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?
I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
- 141
5
votes
2
answers
139
views
What are the common pitfalls that we could face when training neural networks?
Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?
- 51
1
vote
1
answer
80
views
Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?
(link to paper in arxiv)
In section 2.1 the authors define $\gamma$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say:
We ...
- 161
2
votes
2
answers
193
views
How do LSTM and GRU avoid to overcome the vanishing gradient problem?
I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
- 1,433
5
votes
1
answer
897
views
What effect does batch norm have on the gradient?
Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
6
votes
1
answer
745
views
If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem.
The original 2015 paper contains the following passage in section 4.1:
We ...
- 1,259
1
vote
0
answers
341
views
Which activation functions can lead to the vanishing gradient problem?
From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem.
What other activation functions can lead to the ...
- 1,433
5
votes
1
answer
3k
views
How to detect vanishing gradients?
Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how?
For example, if only 25% of my ...
- 153
6
votes
1
answer
8k
views
Why do ResNets avoid the vanishing gradient problem?
I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
- 179