7 votes
Accepted

Why do ResNets avoid the vanishing gradient problem?

Before proceeding, it's important to note that ResNets, as pointed out here, were not introduced to specifically solve the VGP, but to improve learning in general. In fact, the authors of ResNet, in ...
nbro's user avatar
  • 40.5k
6 votes
Accepted

If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?

They explained in the paper why they introduce residual blocks. They argue that it's easier to learn residual functions $F(x) = H(x) - x$ and then add them to the original representation $x$ to get ...
Brale's user avatar
  • 2,376
6 votes
Accepted

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

No, ResNet were not introduced to solve vanishing gradients, citing from the paper: An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which ...
Alberto's user avatar
  • 1,885
5 votes

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

Vanishing gradient problem is indeed present in MLP and CNNs. Please have a look at ResNet paper: before the introduction of residual blocks, the vanishing gradient problem was one of the main ...
Ciodar's user avatar
  • 400
4 votes
Accepted

Why aren't artificial derivatives used more often to solve the vanishing gradient problem?

This idea seems pretty convincing Indeed, you don't have to use the exact gradient of the activation function during the backward step. The gradient of the activation function is a multiplicative ...
Vladislav Gladkikh's user avatar
3 votes
Accepted

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. ...
Victor Björkgren's user avatar
3 votes

What effect does batch norm have on the gradient?

"Naturally, this will affect the gradient through the network." this statement is only partially true, let's see why by starting explaining the real aim of batch normalisation. As the title of the ...
Edoardo Guerriero's user avatar
3 votes
Accepted

How to detect vanishing gradients?

Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0. If only 25% of your kernel weights are changing that does ...
keshav's user avatar
  • 56
2 votes
Accepted

Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can ...
Jzbach's user avatar
  • 136
2 votes

How does vanish gradient restrict RNN to not work for long range dependencies?

Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers ...
amin's user avatar
  • 420
2 votes

What are the common pitfalls that we could face when training neural networks?

There are several pitfalls or issues that require your attention when or before training or using neural networks. I will list some of them below, along with some questions you need to ask yourself ...
nbro's user avatar
  • 40.5k
2 votes

What are the common pitfalls that we could face when training neural networks?

I can't say that it is the biggest problem of deep neural network but it is one of the big problem with deep neural network. Other issue which happens a lot is overfitting on the training data hence ...
overflow95's user avatar
2 votes

Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?

Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd ...
programjames's user avatar
2 votes

Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

Two problems for RNNs They're typically "deeper" than a MLP since you can unroll a RNN into a stack of layers for each timestep, which exacerbates the problem The activation function of ...
mf908's user avatar
  • 21
1 vote
Accepted

How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?

You already mention quite some good ideas. Unfortunately, what makes GANs difficult to work with is their adversarial training paradigm. There are a ton of very simple and intricate ideas which might ...
Robin van Hoorn's user avatar
1 vote
Accepted

How does not learning far inputs make the RNN forget far inputs?

It is actually very simple, it is not about forgetting, but not learning at all means no memorization at all. For long term dependencies, since the gradient is very small, what is learned is very ...
Dr. Snoopy's user avatar
  • 1,355
1 vote
Accepted

Does the paper "On the difficulty of training Recurrent Neural Networks" (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?

It is an error. But it is also not the in final version of the paper (arxiv.) The final version of the paper can be found here where they replace "absolute value of the largest eigenvalue" with "...
Jeremiah England's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible