7
votes
Accepted
Why do ResNets avoid the vanishing gradient problem?
Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in ...
- 38.2k
6
votes
Accepted
If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?
They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get ...
- 2,296
4
votes
Accepted
Why aren't artificial derivatives used more often to solve the vanishing gradient problem?
This idea seems pretty convincing
Indeed, you don't have to use the exact gradient of the activation function during the backward step.
The gradient of the activation function is a multiplicative ...
3
votes
Accepted
How to detect vanishing gradients?
Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0.
If only 25% of your kernel weights are changing that does ...
- 56
2
votes
Accepted
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
- 136
2
votes
How does vanish gradient restrict RNN to not work for long range dependencies?
Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers ...
- 420
2
votes
What are the common pitfalls that we could face when training neural networks?
There are several pitfalls or issues that require your attention when or before training or using neural networks. I will list some of them below, along with some questions you need to ask yourself ...
- 38.2k
2
votes
What are the common pitfalls that we could face when training neural networks?
I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network.
Other issue which happens a lot is overfitting on the training data hence ...
- 188
1
vote
Accepted
How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?
You already mention quite some good ideas. Unfortunately, what makes GANs difficult to work with is their adversarial training paradigm. There are a ton of very simple and intricate ideas which might ...
- 1,574
1
vote
Accepted
How does not learning far inputs make the RNN forget far inputs?
It is actually very simple, it is not about forgetting, but not learning at all means no memorization at all.
For long term dependencies, since the gradient is very small, what is learned is very ...
- 1,122
1
vote
Accepted
Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?
It is an error. But it is also not the in final version of the paper (arxiv.) The final version of the paper can be found here where they replace "absolute value of the largest eigenvalue" with "...
- 161
1
vote
What effect does batch norm have on the gradient?
"Naturally, this will affect the gradient through the network." this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation.
As the title of the ...
- 4,948
Only top scored, non community-wiki answers of a minimum length are eligible