Is the decoder in a transformer Seq2Seq model non parallelizable?

Question

From my understanding, seq2seq models work by first computing a representation of the input sequence, and feeding this to the decoder. The decoder then predicts each token in the output sequence in an autoregressive manner. In this sense, it's limited to processing one time step at a time, as the next token in the sequence depends on the previous (non parallelizable). This is different from the encoder, as it's able to process many time steps in parallel. Is my understanding correct?

Answer 1

During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) :

You input <s> I love you, and the decoder learns to produce I love you </s>. For each token, the model learns to predict the next token (for example given <s> the model learns to predict I). This is possible thanks to attention.

---

But at inference time, you are right : the model has to predict the first token, then put this token in the decoder input and predict the second, etc...

So yes, at inference time a decoder is much slower (because we go through the whole decoder several times : one time for each token) than an encoder (which do a single forward pass).