0
$\begingroup$

Consider an input to RNN $ x = \{x_i\}_{1}^{n}$. Assume that the length of each input $x_i$ is k.

Now, consider the following diagram from p5 of this pdf

enter image description here

My doubts are:

  1. What should I pass as $h_0$? is it a zero vector?

  2. Does RNN updates its weight matrices $U, W, V$ after each token of input $x_i$ ? Or updates after passing all tokens of a particular input $x$?

$\endgroup$
1
  • 1
    $\begingroup$ Here you're asking 2 distinct questions, although of course they are both related to RNNs. Generally, I always suggest that people focus on one main/specific question per post. If you have multiple questions, it's better to ask them in separate posts, and make sure you provide the necessary context to understand them in each of the posts. $\endgroup$
    – nbro
    Jun 5 at 12:47
0
$\begingroup$

I am answering this question based on classical backpropagation through time (BPTT) only.

What should I pass as $h_0$? is it a zero vector?

Yes. We know that $h_{t-1}$ is the footprint of the first $t-1$ tokens of the input sequence. For the first time, we need to pass $h_0$ along with the token $x_1$. Since, footprint is not formed yet, we need to pass zero vector only. Check Figure 9.4 of your pdf.


Does RNN updates its weight matrices $U,W,V$ after each token of input $x_i$ ?

No, RNN does not update its weight matrices $U,W,V$ after each token of input $x_i$. RNN forward pass generates output sequence $y = \{y_i\}_{1}^{n}$ for given input $x$ and the various weight matrices $U, V, W$ are shared across all timestamps of the forward pass.


Or updates after passing all tokens of a particular input x?

Yes, all the weight matrices get updated only after generating the complete $y$.

But since the classical BPTT may take much time and the gradients may vanish gradually, truncated backpropagation through time is generally used. And the updation of weights takes place after generating the partial output only.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.