Can the decoder in a transformer model be parallelized like the encoder?

As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder, this is not possible (in both training and testing), as self-attention is calculated based on previous timestep outputs. Even if we consider some techniques, like teacher forcing, where we are concatenating expected output with obtained, this still has a sequential input from the previous timestep.

In this case, apart from the improvement in capturing long-term dependencies, is using a transformer-decoder better than say an LSTM, when comparing purely on the basis of parallelization?


Can the decoder in a transformer model be parallelized like the encoder?


Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM.

Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute the output. Which makes straight parallelisation impossible.

For detailed explanation of how Transformer works I suggest reading this article: The Illustrated Transformer.

Is using a transformer-decoder better than say an lstm when comparing purely on the basis of parallelization?


Parallelization is the main drawback of RNNs in general. In a simple way, RNNs have the ability to memorize but not parallelize while CNNs have the opposite. Transformers are so powerful because they combine both parallelization (at least partially) and memorizing.

In Natural Language Processing for example, where RNNs are used to be so effective, if you take a look at GLUE leaderboard you will find that most of the world leading algorithms today are Transformer-based (e.g BERT by GOOGLE, GPT by OpenAI..)

For better understanding of why Transformers are better than CNNs I suggest reading this Medium article: How Transformers Work.

| improve this answer | |
  • $\begingroup$ This answer is misleading since it does not mention the fact that the decoder part of a Transformer is parallelizable during training. $\endgroup$ – Mathias Müller Jan 29 at 14:28
  • $\begingroup$ Thanks for the note @MathiasMüller. However, while the decoder can be parallelized during training using the 'already known' trick, to my knowledge this will not have the same result because you can replace by a word that isn't the same as the one the decoder will predict. And this will affect the model differently during back-propagation. So my answer concerns the general understanding of transformer decoder and actual ability to parallelize without a trick that will affect the model. Please clarify if I've mistaken smthg. $\endgroup$ – HLeb May 6 at 13:24
  • $\begingroup$ No, this is not a trick that changes the training procedure: all implementations of standard Transformers compute all positions in the same layer in parallel (for both encoder and decoder during training, for encoder during translation). This does not affect the model at all: the results for each position are mathematically independent. $\endgroup$ – Mathias Müller May 6 at 14:03
  • $\begingroup$ ... In case you meant that during training, actual predictions by the model are not used to build up the target sequence: this is also not a trick I would say, but a standard procedure called "teacher forcing" that is used in virtually all supervised sequence prediction models. $\endgroup$ – Mathias Müller May 6 at 15:34

Can the decoder in a transformer model be parallelized like the encoder?

The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation.

What exactly is parallelized?

Also, it's worth mentioning that "parallelization" in this case means to compute encoder or decoder states in paralllel for all positions of the input sequence. Parallelization over several layers is not possible: the first layer of a multi-layer encoder or decoder still needs to finish computing all positions in parallel before the second layer can start computing.

Why can the decoder be parallelized position-wise during training?

For each position in the input sequence, a Transformer decoder produces a decoder state as an output. (The decoder state is then used to eventually predict a token in the target sequence.)

In order to compute one decoder state for a particular position in the sequence of states, the network consumes as inputs: 1) the entire input sequence and 2) the target words that were generated previously.

During training, the target words generated previously are known, since they are taken from the target side of our parallel training data. This is the reason why computation can be factored over positions.

During inference (also called "testing", or "translation"), the target words previously generated are predicted by the model, and computing decoder states must be performed sequentially for this reason.

Comparison to RNN models

While Transformers can parallelize over input positions during training, an encoder-decoder model based on RNNs cannot parallelize positions. This means that Transformers are generally faster to train, while RNNs are faster for inference.

This observation leads to the nowadays common practice of training Transformer models and then using sequence-level distillation to learn an RNN model that mimicks the trained Transformer, for faster inference.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.