1
$\begingroup$

I am aware that back-propagation through time is used for training the recurrent neural network. But I am not able to understand how this happens for the bi-directional versions of the recurrent neural networks?

So, I was hoping if anyone help me with:

  1. Understanding with an example the training of bi-directional recurrent neural networks using back-propagation through time? (I tried following the original paper https://ieeexplore.ieee.org/document/650093, but it was kind of confusing for me when they perform the backward pass for training)
$\endgroup$
1
$\begingroup$

I have not implement the backprop of a bi-directional RNN from scratch so I can't be sure my answer is correct but I hope it helps.

You can see how bi-directional RNN works from this video from Andrew NG. I got the image below from that video: enter image description here

For more clarity: enter image description here

So if you know how to backprop through a simple RNN, you should be able to do so for bi-directional RNN.

If you need more detail, let me know.

$\endgroup$
6
  • $\begingroup$ Thank you @amin. I have one more doubt as to if I decide to fetch output from intermediate layers of the bi-directional GRU ( which is in this case the same as hidden state as GRU outputs only the hidden state at any time 't', how do i extract the weights of such a layer which I want to fetch? $\endgroup$ Oct 19 '20 at 13:17
  • 1
    $\begingroup$ Unfortunately I can't understand what you wrote well. But I will tell you 2 notes and I hope they help: 1- if you are using keras, you can set the return_sequences parameter of your GRU layer to true, and then when using GRU you will get a list of outputs containing output of each timestep. 2- RNN will use just one bunch of weights for all timesteps (so the same weights are applied in each timestep) and BRNN will use two bunch of weights, one for forward (going from x<1> to x<4>) and one for backward (going from x<4> to x<1> in the picture above) $\endgroup$
    – amin
    Oct 20 '20 at 2:54
  • $\begingroup$ Thank you @amin. I will just like to add one more scenario to the one you suggested in the second point. Suppose I stack BRNN layers in a model I am trying to build. For ex- if no. of stacked layers are 5, and I want to extract intermediate outputs of BRNN layer numbers- 2, 3, 4 apart from ( with the inputs fed to layer 1 of BRNN). I am using Keras and Functional API to implement. In this case, as I am extracting outputs from inner BRNN layers, will the inner connections of my network differ from a normal BRNN? Moreover how do i extract intermediate weights when utilizing backpropagation? $\endgroup$ Oct 20 '20 at 4:29
  • 1
    $\begingroup$ @rahultomar I'm glad if I helped. I prefer to give you a useful links to understand how RNNs work in keras and how to use it. I guess if you understand what has been said in this link, you can answer the questions above. If you can't get your answer by looking at this link, I suggest you to have a post in stackoverflow (as ai.stackexchange is not related to the programming problems). Keras - Working with RNNs. Good luck and have fun programming $\endgroup$
    – amin
    Oct 20 '20 at 6:53
  • 1
    $\begingroup$ Thank you @amin for your response. As suggested i will post on the required forum. Meanwhile, coming to the same weights (one for forward and another for backward which are shared for the BRNN). Does this hold true as we stack BRNN layers on top of each other? More specifically, if W1 are the set of weights in the forward RNN of first BRNN layer while W2 are weights of backward RNN of first layer. As both W1 and W2 hold for all timesteps in the same layer, do they change as we stack up more layers or they remain same ? $\endgroup$ Oct 20 '20 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.