# Why is the vanishing gradient problem especially relevant for a RNN and not a MLP

I would like to know why the vanishing gradient problem especially relevant for a RNN and not a MLP (multi-layer-pereptron). In a MLP you also backpropagate errors and multiple different weigths. If the weights are small, the resulting update in the last layers in the backpropagation will be very small

• In the most simple case, you can think of RNN as a recursive MLP that calls itself multiple times. We can unfold it into a very deep MLP and as a result, the gradients vanish as they propagate through the numerous layers. Jan 9 at 16:57

No, ResNet were not introduced to solve vanishing gradients, citing from the paper:

An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning. This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].

However, vanishing gradient happens also for MLP for the same reasons why they happen in RNNs as you can see an unrolled RNN as a MLP at the end of the day: because you stack multiple layer, and if many of them saturate, the gradient will tend to zero

You can see it from an unrolled RNN:

Here, the gradient of $$E_4$$ with respect to $$x_0$$ will have to travel 6 matrix multiplications/non linearities, even though the net is just 1 layer deep.

If the spectral norm of such matrices is less than one (ie the highest eigenvalue is < 1), then they are contractive, and thus a vector multiplied many times by such matrix will be attracted to 0

• Thanks Alberto for your answer. What I still don't understand is why are RNN especially prone for the vanishing gradient problem? I mean if you use a deep MLP it will look similar as in your figure and why is a deep MLP not prone to the vanishing gradient problem as strong as a RNN? Jan 10 at 8:28
• @PeterBe you should be able to see it from the picture. the problem is that the "connection" from the first input (x0) and the last output (En) is as long as the length of the input size... thus even a network with 1 recurrent layer, given a string of length 50, the last output is 50 layer transformation away from the first input (thus is like having a 50 layers deep net) Jan 10 at 10:18
• @PeterBe you're welcome, good luck with your journey Jan 10 at 15:04

Vanishing gradient problem is indeed present in MLP and CNNs. Please have a look at ResNet paper: before the introduction of residual blocks, the vanishing gradient problem was one of the main limitations to the depth of the networks. In RNNs this problem is present also with fewer layers because each layer needs to perform backpropagation through time, so learning is additionally constrained by the sequence length.

• Thanks for your answer. What I don't really understand is why are RNN especially prone for the vanishing gradient problem? So why is the backpropagation through time as you wrote, especially prone to that? Jan 9 at 16:53

Two problems for RNNs

1. They're typically "deeper" than a MLP since you can unroll a RNN into a stack of layers for each timestep, which exacerbates the problem
2. The activation function of choice typically is tanh which has diminishing gradients at the extremes
• Thanks mf908 for your answer. Let's say I have a deep MLP and I also use tanh as the activiation function. Would it be as prone to the vanishing gradient problem as a RNN? Jan 10 at 8:29
• @PeterBe The deeper the network, the worse it's going to be. As far as the vanishing gradient issue is concerned, an RNN of k layers over n timesteps is equivalent to a MLP of kn layers. Can you see that this number kn (k times n) can easily be very large? Of course it's theoretically possible to come up with a very deep MLP and a very shallow RNN over very few timesteps, and then you can say "the vanishing gradient issue is worse for this MLP than for this RNN!!", but in general it's much worse for RNNs than for MLP.
– Stef
Jan 10 at 13:37
• Theoretically yes, it's a part of the reason why ReLu is the activation function of choice nowadays, previously it was sigmoid which similarly has diminishing gradient issues Jan 10 at 21:54