1
$\begingroup$

If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step.

We know that the actual outputs cannot be given to the model while testing, then what information passes from a time step to the next time step in the test phase?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't think there's any difference between making predictions when you use or not teacher forcing during training. So, let me describe one way of doing that.

During testing, as you noticed, you don't know the ground-truth labels, so the only way of predicting a sequence is to feed into the model the predictions for the previous time-steps.

So, let's say that your model is denoted by $f$, which attempts to approximate the language model $p(x_t \mid x_{1:t-1})$, a conditional probability distribution over characters.

To perform inference, you first need to provide the initial character of the sequence, usually, a special character, like $\langle \text{start} \rangle$, so your model predicts the first real character of the sequence as follows $$f([\langle \text{start} \rangle]) = \hat{x}_1.$$ Afterward, you feed to it $\hat{x}_1$, in order to produce the next character, i.e. $$f([\langle \text{start}\rangle, \hat{x}_1]) = \hat{x}_2,$$ and so on, until the model $f$ produces another special character that denotes the end of the sentence $\langle \text{end}\rangle$.

Now, in practice, rather than all the previously predicted characters, you may just feed the character $x_{t-1}$ to predict $x_t$, and, at the same time, pass the previous last state of the RNN, so something like $$f (\hat{x}_{t-1}, h_{t-1}) = \hat{x}_t.$$

The following diagram illustrates the concept.

enter image description here

Here the first char is $T$ and the model predicts that the next char is $e$. Then we pass $e$ and the previous state of the model to the model to predict $n$, and so on.

Here you have a TensorFlow example that generates text with an RNN, uses teacher forcing, and then performs inference in the way I described (of course, there are other details, like using a temperature).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Then the architecture of RNN is changing, right? $\endgroup$
    – hanugm
    Commented Dec 23, 2021 at 10:44
  • 1
    $\begingroup$ @hanugm No, the architecture doesn't change, at least, not usually. What changes is what you pass as input to the model. The hidden state of a NN can be thought of as containing the information that would allow you to make a prediction based on the previous inputs or history, at least, this is the intuition. If you look at the linked code (TF example), maybe you will get a better sense of what's happening. $\endgroup$
    – nbro
    Commented Dec 23, 2021 at 11:13
  • $\begingroup$ In that example, basically, you can make the RNN (in that case, a GRU) return you the last state of that layer after having taken the input. Then you use that last state as some kind of "context vector" for predicting the next char, and that GRU produces another "context vector" (i.e. the hidden vector), which you use to make a prediction at the next time step, and so on. $\endgroup$
    – nbro
    Commented Dec 23, 2021 at 11:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .