Initial Input $h_0$ for RNN and updation of weights

Question

Consider an input to RNN $ x = \{x_i\}_{1}^{n}$. Assume that the length of each input $x_i$ is k.

Now, consider the following diagram from p5 of this pdf

My doubts are:

1. What should I pass as $h_0$? is it a zero vector?

2. Does RNN updates its weight matrices $U, W, V$ after each token of input $x_i$ ? Or updates after passing all tokens of a particular input $x$?

Answer 1

I am answering this question based on classical backpropagation through time (BPTT) only.

What should I pass as $h_0$? is it a zero vector?

Yes. We know that $h_{t-1}$ is the footprint of the first $t-1$ tokens of the input sequence. For the first time, we need to pass $h_0$ along with the token $x_1$. Since, footprint is not formed yet, we need to pass zero vector only. Check Figure 9.4 of your pdf.

---

Does RNN updates its weight matrices $U,W,V$ after each token of input $x_i$ ?

No, RNN does not update its weight matrices $U,W,V$ after each token of input $x_i$. RNN forward pass generates output sequence $y = \{y_i\}_{1}^{n}$ for given input $x$ and the various weight matrices $U, V, W$ are shared across all timestamps of the forward pass.

---

Or updates after passing all tokens of a particular input x?

Yes, all the weight matrices get updated only after generating the complete $y$.

But since the classical BPTT may take much time and the gradients may vanish gradually, truncated backpropagation through time is generally used. And the updation of weights takes place after generating the partial output only.