I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input to the output of masked multi-head attention block adds some of the information from future time steps which is then used to construct the query matrix for multi-head attention block.

Shouldn't some kind of mask be applied before adding the input values to the output of masked multi-head attention block as well?

Asking this question in the context of training process.

Reference Image for Decoder Architecture

  • $\begingroup$ What do you think the output of Masked (Multi-Head) Attention, if the input is "I am a boy"? $\endgroup$
    – Cloud Cho
    Commented Nov 16, 2023 at 22:57
  • $\begingroup$ The output would comprise of 4 vectors, each corresponding to one of the words in input sequence (assuming tokenization is word based). The output vector corresponding to word "am" will only be determined by the embeddings corresponding to "I" and "am", no information about "a" and "boy" is used for its computation. Masking is done by adding upper-triangular matrix with all values equal to $-\infty$ before applying softmax to obtain attention weights. Multi-Head just means that several attention blocks will be used, output from which will be concatenated and mapped back to same dimensions. $\endgroup$ Commented Nov 18, 2023 at 9:47
  • $\begingroup$ If the output of four vectors, could you guess how to add the input to the output (and normalized)? $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 16:45

1 Answer 1


No, adding an additional mask to the residual connections or the linear layer isn't necessary. The masking is crucial solely for the attention mechanism. Implementing a mask in the residual connections or the linear layer would only reduce dimensionality and overly constrain the model's outputs. Remember, for the final output, it's essential that the model has full contextual information. Zeroing a value at any step would make the model less efficient by causing it to lose information.

It's important to note that even within the attention mechanism, the input data is already being utilized in the attention computation. This is evident after the masking of the attention weights, where the matrix $QK^T$ is projected again with the $V$ matrix, which is equal to $W_VX$. Where $X$ is the input data. Thus, worrying about retaining the mask output is unnecessary. Additionally, when multiplying with the $V$ matrix, you're already losing the lower triangular matrix. From that point on, adding the initial input doesn't pose an issue.

Another crucial aspect to consider is the significance of the residual layers. Without them, the positional information from the Positional Encoding (PE) would rapidly diminish, making it vital to retain these layers. So, even if, for some reason, it affects the property of not seeing future tokens (which it doesn't), you would still want to maintain the residual connections.

  • $\begingroup$ Regarding "Positional Encoding (PE) would rapidly diminish", will it be happened in this neural network model? This model has only two forward layers not around 20 layers? $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 16:47
  • $\begingroup$ In a certain degree it will still happen, and the residual connections in the model evidence that. Another reason for the Residuals is the vanishing gradient effect. They are still needed for solving this issue if it only has 2 forward layers? Probably not as necessary for 20 layers but it can help. Another consideration is that technically speaking very few implementations use the full Transformer architecture. Most language models use BERT or GPT that are stacked multi-head attention layers, so in practice Transformers do use a bunch of layers. $\endgroup$
    – Cesar Ruiz
    Commented Nov 20, 2023 at 20:47
  • $\begingroup$ Thanks for aspect of application of Transformer. Is there any reference we could see entire model structure of BERT or GPT? When I search in Google, BERT looks like 12 to 24 transformers from researchgate.net/figure/…. For me, adding, skipping layer, at 24 layers is still questionable. Have you thought that Masked Attention isn't actually not working? $\endgroup$
    – Cloud Cho
    Commented Nov 20, 2023 at 22:10
  • $\begingroup$ I understood the implementation of BERT and GPT with these repos nanoGPT and BERT-pytorch. In general, both architectures can be scaled indefinitely, there is no limit to how many transformer blocks you can use, so 24 is really an arbitrary number, it can be any number. For the other question, I did not find it questionable to add the skip connections, it is a very common practice. Then Masked Attention not working would depend on the objective to solve. This helps the model process information with human-like reasoning (because humans can't process language in chunks but as a sequence) $\endgroup$
    – Cesar Ruiz
    Commented Nov 21, 2023 at 0:30
  • $\begingroup$ "...add the skip connections, it is a very common practice" I actually haven't seen any other with the masked layer. All other neural network models seem like not having masking operation. If you find any, please share. $\endgroup$
    – Cloud Cho
    Commented Nov 23, 2023 at 9:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .