# Masking in Decoder of Transformer

I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block.

Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well?

Asking this question in the context of training process.

• What do you think the output of Masked (Multi-Head) Attention, if the input is "I am a boy"? Commented Nov 16, 2023 at 22:57
• The output would comprise of 4 vectors, each corresponding to one of the words in input sequence (assuming tokenization is word based). The output vector corresponding to word "am" will only be determined by the embeddings corresponding to "I" and "am", no information about "a" and "boy" is used for its computation. Masking is done by adding upper-triangular matrix with all values equal to $-\infty$ before applying softmax to obtain attention weights. Multi-Head just means that several attention blocks will be used, output from which will be concatenated and mapped back to same dimensions. Commented Nov 18, 2023 at 9:47
• If the output of four vectors, could you guess how to add the input to the output (and normalized)? Commented Nov 20, 2023 at 16:45

It's important to note that even within the attention mechanism, the input data is already being utilized in the attention computation. This is evident after the masking of the attention weights, where the matrix $$QK^T$$ is projected again with the $$V$$ matrix, which is equal to $$W_VX$$. Where $$X$$ is the input data. Thus, worrying about retaining the mask output is unnecessary. Additionally, when multiplying with the $$V$$ matrix, you're already losing the lower triangular matrix. From that point on, adding the initial input doesn't pose an issue.