The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like

self.deltas[x] = self.deltas[x+1].dot(self.weights[x].T) * self.layers[x] * (1- self.layers[x]) where

self.deltas[x+1] is error from prevous layer, self.weights[x] is weights map and self.layers[x](1- self.layers[x]) is bakwards activation of sigmoid function where self.layers[x] is vector of sigmoid. But while normal backpropagation the values are there, while BPTT i can not take the current self.layers[x] : i need the previous ones, right ?

So unlike normal BP, do i need extra store old weights and layers, for example in circular queue, and then apply the formula where self.deltas[x+1] is layer from next time ?

Not realy implementation, just basic understanding in order to can implement it.

Lets see the picture:

enter image description here

Here are : self.layers[0] = $x_{t+1}$, self.layers[1] = $h_{t+1}$ , self.layers[2] = $o_{t+1}$, in order to perform backprop $h_{t+1}$ -> $h_{t}$ -> $h_{t-1}$... I DO NEED to have layers $h_t$ ,$h_{t-1}$... and weights $v_{t+1}$, $v_t$... EXTRA stored in additional to the network $x_{t+1}$ -> $h_{t+1}$ -> $o_{t+1}$, right? Thats all the question.

And i do not need to store previous outputs $o[t, o_{t-1}, etc..]$, because backprop from them ot->ht, etc was already calculated.


One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

Notation of my explanation is different from notation in the figure of question. My notation is as below:

  1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
  2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
  3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed slightly different notation than in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/enter link description here



Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

  • $\begingroup$ Wait a second, its confusin again. That means, everything above ht must also be recalculated, so i store also 'U'-s and also make sum it each time? W does not have chain from V, so does not need to be stored, eight? $\endgroup$ – user8426627 Jun 12 '19 at 11:22
  • $\begingroup$ I didn't understand what you mean by everything above ht must also be recalculated. Calculation of gradients of both Us and Ws will have time (summation over time). $\endgroup$ – Kartik Podugu Jun 12 '19 at 12:09
  • $\begingroup$ why W is also summed over time? It is not recurrent and gaings no information from U. if you look seperatels just h->o ther ewill be just 'normal' BP. $\endgroup$ – user8426627 Jun 12 '19 at 12:16
  • $\begingroup$ I am sorry. Since you also used U, V, W; i thought notation is same. That is where the confusion came. U --> Input Layer - Hidden Layer V --> Hidden Layer - Output Layer W --> Hidden Layer - Hidden Layer According to my convention V is independent of previous time steps. U, W depend on previous time steps. But, as per your convention in the above figure in the question, W is independent of previous time steps. U and V depend on previous time steps. $\endgroup$ – Kartik Podugu Jun 13 '19 at 11:45
  • $\begingroup$ reread my answer, i have updated notation details and another youtube link which has clear derivation details. I felt video is better than deriving here. $\endgroup$ – Kartik Podugu Jun 13 '19 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.