If my dataset consists of the sentence "I have an apple" do I need to feed my model the separate examples "I" -> "have", "I have" -> "an", and "I have an" -> "apple" or is that essentially what the masking does, and I only have to feed in the entire sentence at once?


1 Answer 1


If you are training a decoder-only model, you don't need any manual augmentation. Recall that a decoder-only model only allows tokens to attend to previous tokens during self-attention. Additionally, decoder-only models will often predict the next token at the position of the next-to-last token. In other words, the model's output representation at position $i$ (say, $h_i$) is used to predict the token $t_{i+1}$, with $h_i$ only depending on tokens $t_1, ..., t_{i-1}$.

So, with a single forward pass, each output hidden state $h_i$ (and subsequent token prediction) constitutes a valid autoregressive prediction (i.e., it predicts the next token based on only the previous values).

Contrast that with training an encoder-only MLM model, like BERT, where (part of) the pretraining objective is to predict a masked-out word. This masking can be in the form of a [MASK] token or a randomly chosen token. You have to do some kind of masking as otherwise the model will just learn to repeat the input token (although in the BERT paper, they do leave the token unchanged 10% of the time). To produce these masks, you need to do some preprocessing beforehand, although you can mask multiple tokens (and thus calculate losses for them) in a single forward pass.

Encoder-decoder models are often also trained a similar way. E.g., for T5, you mask out, then predict consecutive spans of tokens, but your model can predict multiple token spans per forward pass.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .