I am trying to understand the purpose of masking in the decoder in the "Attention is All you Need" paper. Why wouldn't we want to rely on bidirectional context when translating? What goes wrong if masking isn't used in the decoder? Are there successful models in which the decoder doesn't use masking? Thank you!
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Masking
The purpose of masking in the decoder is to prevent the model from having access to future tokens when predicting the current token in the output sequence. During training, the model is given the entire target sequence as input, but it should not be allowed to use information from future tokens to predict the current token, as this would violate the autoregressive nature of the decoding process.
The decoder should only have access to the leftward (previous) context when generating a translation. If masking were not used, the model would be able to "cheat" by leveraging information from future tokens, which would not be available during inference or when generating translations in a real-world scenario. This would lead to a model that performs well on training data but fails to generalize to new, unseen data.
Alternative Approaches
As for models that do not use masking in the decoder, most sequence-to-sequence models rely on some form of autoregressive decoding, either through masking or through other mechanisms. For example, traditional RNN-based encoder-decoder models do not use explicit masking, as they generate the output sequence one token at a time, inherently relying only on the previous context. However, there are non-autoregressive models that predict the entire output sequence in parallel, which do not use masking in the same way as autoregressive models. These non-autoregressive models can be faster but often sacrifice some accuracy compared to their autoregressive counterparts.
