How do we train a seq2seq rnn training?

We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient descent by comparing them with the actual word expected. Then we input to the second decoder the hidden state and it outputs the second word with probabilities. We do another gradient descent. But what if the first hidden state was wrong because it failed to output the right word? Then this second gradient descent is meaningless?


1 Answer 1


In seq2seq RNN training, we usually use a technique called "teacher forcing." With teacher forcing, the actual (ground truth) output word at each timestep is fed as input to the decoder in the next timestep, rather than using the model's previous prediction. This approach helps the model learn faster because it doesn't rely on previous potentially incorrect predictions, which might indeed lead to a cascading effect of errors.

However, during inference (when actually translating sentences), the model doesn't have access to ground truth outputs. So, it uses its own predictions as input for the next timestep. This can lead to some discrepancies between training and inference, known as "exposure bias." To mitigate this, a technique called "scheduled sampling" is sometimes used, which involves gradually transitioning from teacher forcing to using the model's own predictions during training.

  • $\begingroup$ How can we do teacher forcing when the things decoder pass to themselves are hidden states? Do we need to change what they pass themselves (maybe just the last hidden state of the encoder and the raw words output)? $\endgroup$ Mar 21, 2023 at 8:41
  • $\begingroup$ You don't need to change what the decoder passes to itself. Instead, you alter the input to the decoder during training. $\endgroup$ Mar 21, 2023 at 8:59
  • $\begingroup$ During training you input words instead of hidden states to the decoder? $\endgroup$ Mar 21, 2023 at 9:26
  • $\begingroup$ Oh my bad, you keep the same hidden state but you input the correct word + the hidden state instead of the wrong word + the hidden state? $\endgroup$ Mar 23, 2023 at 10:28
  • 1
    $\begingroup$ Yes, that's correct. During training with teacher forcing, you keep the same hidden state updating mechanism, but at each time step of the decoder, you input the correct (ground truth) word/token from the target sequence along with the hidden state, instead of using the predicted word/token from the previous time step. $\endgroup$ Mar 23, 2023 at 10:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .