Return to Answer

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed the notation in the first link, slightly different than notation in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]2

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

Notation of my explanation is different from notation in the figure of question. My notation is as below:

1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

Notation of my explanation is different from notation in the figure of question. My notation is as below:

1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed slightly different notation than in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]2

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed your notation.

Some links that will help you explore.

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]1

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

Notation of my explanation is different from notation in the figure of question. My notation is as below:

1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed the notation in the first link, slightly different than notation in the diagram in question.

Some links that will help you explore.

https://www.youtube.com/watch?v=RrB605Mbpic (clearly explains about gradients of all three U, V, W; but notation is different from diagram in question)

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]2

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

Notation of my explanation is different from notation in the figure of question. My notation is as below:

1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed your notation.

Some links that will help you explore.

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]1

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

One word answer for your question "Do you need to store previous values of weights and layers on recurrent layer while BPTT?" is YES

Let us go through the details.

For training an RNN using BPTT, we need gradients of error w.r.t all three parameters U, V, W

And for calculating these gradients, we use chain rule of differentiation, the same rule that we used to calculate gradients in a fully connected neural network.

The gradient w.r.t V only depends on current time step (doesn't need any values from previous time step).

The gradients w.r.t U, W depends on current time step and also all previous time steps (so needs values from all time steps)

Basically, we need to back propagate gradients from current time step all the way to t=0.

How this back propagation is different from the back propagation we use in fully connected neural network is that, in fully connected neural network we don't have the concept of t and also we don't share any weights across layers. But, here we share weights across layers and time instants. So, gradients depend on all time instants.

Note: Be careful with notation difference between several articles. I followed your notation.

Some links that will help you explore.

http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/[enter link description here]1

http://ir.hit.edu.cn/~jguo/docs/notes/bptt.pdf

https://www.d2l.ai/chapter_recurrent-neural-networks/bptt.html

Remember, you should understand chain rule of partial derivative very clearly to do the derivation yourself and understand it.

Also, dont think BPTT is separate than BP. It is one and the same. Since neural network architecture in RNN includes time instants and sharing of weights across time instants, just using chain rule on this network makes back propagation also dependent on time and so is the name.

Hope it helps. Feedback is welcome.

Notation of my explanation is different from notation in the figure of question. My notation is as below:

1. V - Hidden Layer - Output Layer (gradients of V are independent of previous time steps)
2. U - Input Layer - Hidden Layer (gradients of U are dependent on previous time steps)
3. W - Hidden Layer - Hidden Layer (gradients of W are also dependent on previous time steps)