Transformer networks are great, because they can handle variable length inputs, but they also have limitations, concerning the input size. For example BERT (a transformer based language model) only accepts $N = 512$ input tokens at most. They way transformer models can accept sequences shorter than the $N$, is by padding the input sequence with zeros and masking. When inputting a sequence with length $n \leq N$, the mask $m$ is a vector of size $N$, where $m_i = (i \leq n)$. Say we have a transformer that should translate from English to German which accepts $N = 8$ tokens at most, and we input the sentence "Today is Tuesday". Let the words have the following token values:
\begin{array} {|r|r|}
\hline \text{[SOS]} & 1 \\
\hline \text{Today} & 274 \\
\hline \text{is} & 12 \\
\hline \text{Tuesday} & 125 \\
\hline \text{[EOS]} & 2 \\
\hline \end{array}
The first input will always be occupied by the [SOS] token and each sequence is followed by an [EOS] token.
Now the input looks like this:
\begin{bmatrix}1\\274\\12\\125\\2\\0\\0\\0\end{bmatrix}
And you also supply an attention mask (using integers representing booleans) which looks like this:
\begin{bmatrix}1\\1\\1\\1\\1\\0\\0\\0\end{bmatrix}
This means, each position can only attend to positions $i$ where $m_i = 1$. Therefore, positions where the mask is 0, do not affect the output of the model.
The vector representation of the mask above is the simplified version of the actual attention mask, describing that position at row $i$ can attend to positions at column $j$:
\begin{bmatrix}1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\end{bmatrix}
The decoder works a bit differently during training and inference.
Training:
The decoder attention mask is built, so that each position can not attend to future positions. You can describe this by using a matrix:
\begin{bmatrix}1&0&0&0&0&0&0&0\\1&1&0&0&0&0&0&0\\1&1&1&0&0&0&0&0\\1&1&1&1&0&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&1&0&0\\1&1&1&1&1&1&1&0\\1&1&1&1&1&1&1&1\end{bmatrix}
When using this mask for the decoder, position $i$ can only attend to positions $j, j \leq i$.
For the task of translation from English to German (e.g. "Today is Tuesday" -> "Heute ist Dienstag"), you can provide the model with the entire input sentence and the entire output sentence. The model will make a prediction for each decoder input position, what the next word will be. And by using the mask, no information leaks into earlier positions (when predicting the next token for the word "ist", the model can only see "[SOS] Heute ist").
Inference:
For the inference stage, you would first provide the input sentence for the encoder and for the decoder only the [SOS] token. Now you sample a word from the transformer output (e.g. picking the most likely or sample from the output distribution) at the position of your last decoder input (in this case the position of the [SOS] token, e.g. position 0). Then you do the next inference step, by again providing the input sentence for the encoder, and for the decoder you provide the [SOS] token + the predicted word from the last step. If the previous prediction was "Heute", your decoder input now is "[SOS] Heute". You just repeat this process, each time providing the so far predicted sentence to the decoder, until the transformer predicts [EOS].
