I'm currently studying the Transformer model (Attention is all you need) and after reading it I still have some questions for which I get conflicting answers if I google them:

  • What exactly are the dimensions of the input to the encoder of a transformer, from what I've seen you can input sentences with dynamic lengths but in the paper it seems like all layers expect the K/Q/V matrices to have dimensions of d_model, d_k or d_v/d_q
  • Same question for the decoder, what exactly are the input dimensions and how do the attention layers handle dynamic dimensions if its possible
  • Another question is is masking only used for training where you input the whole sentence as the input to the decoder or does it have another purpose than that (especially in inference)

1 Answer 1


Transformer networks are great, because they can handle variable length inputs, but they also have limitations, concerning the input size. For example BERT (a transformer based language model) only accepts $N = 512$ input tokens at most. They way transformer models can accept sequences shorter than the $N$, is by padding the input sequence with zeros and masking. When inputting a sequence with length $n \leq N$, the mask $m$ is a vector of size $N$, where $m_i = (i \leq n)$. Say we have a transformer that should translate from English to German which accepts $N = 8$ tokens at most, and we input the sentence "Today is Tuesday". Let the words have the following token values:

\begin{array} {|r|r|} \hline \text{[SOS]} & 1 \\ \hline \text{Today} & 274 \\ \hline \text{is} & 12 \\ \hline \text{Tuesday} & 125 \\ \hline \text{[EOS]} & 2 \\ \hline \end{array}

The first input will always be occupied by the [SOS] token and each sequence is followed by an [EOS] token. Now the input looks like this:


And you also supply an attention mask (using integers representing booleans) which looks like this:


This means, each position can only attend to positions $i$ where $m_i = 1$. Therefore, positions where the mask is 0, do not affect the output of the model.

The vector representation of the mask above is the simplified version of the actual attention mask, describing that position at row $i$ can attend to positions at column $j$:


The decoder works a bit differently during training and inference.

Training: The decoder attention mask is built, so that each position can not attend to future positions. You can describe this by using a matrix:


When using this mask for the decoder, position $i$ can only attend to positions $j, j \leq i$. For the task of translation from English to German (e.g. "Today is Tuesday" -> "Heute ist Dienstag"), you can provide the model with the entire input sentence and the entire output sentence. The model will make a prediction for each decoder input position, what the next word will be. And by using the mask, no information leaks into earlier positions (when predicting the next token for the word "ist", the model can only see "[SOS] Heute ist").

Inference: For the inference stage, you would first provide the input sentence for the encoder and for the decoder only the [SOS] token. Now you sample a word from the transformer output (e.g. picking the most likely or sample from the output distribution) at the position of your last decoder input (in this case the position of the [SOS] token, e.g. position 0). Then you do the next inference step, by again providing the input sentence for the encoder, and for the decoder you provide the [SOS] token + the predicted word from the last step. If the previous prediction was "Heute", your decoder input now is "[SOS] Heute". You just repeat this process, each time providing the so far predicted sentence to the decoder, until the transformer predicts [EOS].

  • $\begingroup$ thanks so much this cleared out a lot of questions i had. One last question, does this mean that for the first step in inference you also input a (n_vocab, n_model) dimensional matrix to the decoder with a an attention mask with just a 1 at the first posiiton (for the first [SOS] token) and then for each step you expand the mask? $\endgroup$ Mar 8 at 23:08
  • $\begingroup$ For inference you also just use the lower left triangular matrix as mask, just like in the training stage. This mask makes sure each prediction can't use future positions and due to the nature of it, it "expands" automatically. You could think that it would make more sense if e.g. pos 2 could also attend to pos 4 after predicting the first 5 words, but as during training this was not possible, changing this for the inference stage would not make sense. I think this is also a slight downside to transformers, as information in the decoder flows unidirectional. Glad I was able to help :) $\endgroup$
    – Evator
    Mar 9 at 1:27
  • 1
    $\begingroup$ Thanks so much, I wish I could upvote, I'll come back to this as soon as I get 15 reputation ^^ $\endgroup$ Mar 9 at 8:24
  • $\begingroup$ Would it be possible to generate an entire German sentence in one go, in multiple passes, by repeatedly feeding back the previous iteration of the sentence into the transformer, with a square/rectangular mask? Would that be useful? Then, the transformer would not be constrained to generate the output one word at a time. $\endgroup$
    – user253751
    Mar 23 at 11:59
  • $\begingroup$ I think what you mean would be equivalent to doing one word at a time/ that is what you are doing during the inference stage of a transformer. If I understand you right you just want to embed this process into the network architecture? $\endgroup$
    – Evator
    Mar 24 at 13:03

Not the answer you're looking for? Browse other questions tagged .