2
$\begingroup$

I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like:

Consider trying to predict the last word in the text “I grew up in France… I speak fluent French.” Recent information suggests that the next word is probably the name of a language, but if we want to narrow down which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large. Unfortunately, as that gap grows, RNNs become unable to learn to connect the information.

it comes due to vanish gradient problem. However, I could not understand that how to vanish gradient creates an issue for RNN to not work for long-range dependencies. Since, as I know that vanish gradient usually comes when we have many hidden layers and the gradient for the first layer usually produced too low and that affects the training process. However, everyone connects this issue with vanish gradient, technically what is the relationship RNN (long-range dependencies) with vanish gradient?

I am really sorry if it is a weird question

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Vanishing gradient is: as the gradient starts to flow from the end of the network (right side of the network) to the start of the network (left side of the network), it will be multiplied by numbers less than 1 and gradually it will become weaker and weaker and when it arrives to the first layers, it's so weak that makes almost no change in initial layers parameters.

Now in case of RNNs you can unroll the network, and now you can see that it is like a deep network. For clarity look at the image below (taken from a course by Andrew NG and edited):

enter image description here

Red arrows show the way of gradient backpropagation and you can see that in each step they are multiplied by a number (actually a matrix is multiplied in another matrix). If this number is less than one, then it results in vanishing gradient. If this number is greater than one it will result in exposure (which can be controlled by simply clipping it to a max value). The higher the steps, the more of vanishing or exploding effect.

But they have found a solution for this problem, LSTM or GRU. What these units do is that they will make a highway for gradient to backpropagate through (in each state it will be multiplied by 1). So gradient can travel for longer distance (from French into France). Even though, the problem still exists for very long relations.
If you know what is a ResNet and how it works, you can find the same concept behind LSTM or GRU and ResNets as both make a highway for gradient to flow back.

You can have an intuition of how LSTM or GRU works by following the forward pass and assuming what they are doing as locking the concepts in memory cells. Like when the network sees the word France it will understand that it's an important word and maybe it will become handy later. So it will put the France word in one of it's memory cells and keeps it there for several steps and when it wants to guess the language it will use this memory cell to predict French not English or Persian. Then it can release that memory cell and use this cell for something else.

If you want to learn more and get better intuition, I highly recommend you to look at this link: CS231n of MIT - lecture 10

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you so much for your answer. I have got somehow, but still, there are some fogs/confusions in my mind. Do you have any resource for RNN, LSTM, GRU where I can find step by step calculation (Calculation on each neuron and weight calculation) on an example, such as this link [mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/…, but it is a simple example on traditional NN and I need for LSTM, RNN and GRU. $\endgroup$
    – Nafees Ahmed
    Oct 22, 2020 at 5:24
  • $\begingroup$ Secondly, I have got the following lines from this [towardsdatascience.com/…, which are as follow as, So in recurrent neural networks, layers that get a small gradient update stops learning. Those are usually the earlier layers. So because these layers don’t learn, RNN’s can forget what it has seen in longer sequences (what is the logic in between this), thus having a short-term memory. $\endgroup$
    – Nafees Ahmed
    Oct 22, 2020 at 5:49
  • $\begingroup$ To your first comment: You can find a simple RNN implementation here. If you have problem to understand the code in the link, make sure you understand what is RNNs by looking the video in the answer above, and if you have time you can see the related videos on coursera, taught by Andrew NG (understanding the concept worth the time). I have no link for LSTM and GRU but you can implement them if you understand them. $\endgroup$
    – amin
    Oct 22, 2020 at 6:24
  • $\begingroup$ To your second comment: Long dependency means the (RNN) network should keep the information for longer time (the network expands) and It's impossible for a simple RNN to learn these long dependencies as it's impossible for a very deep network to learn. Maybe you are still confused, but unfortunately I don't know how to explain it in a few lines $\endgroup$
    – amin
    Oct 22, 2020 at 6:34
  • 1
    $\begingroup$ @amin If we unfold the computational graph of an RNN the weights still get updates (since they are the are reused at every timestep). The problem is not that weights don't get update, the problem is that the contributions to the updates from the earlier layers is negligible. $\endgroup$
    – ado sar
    Jun 20 at 16:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .