** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components within it. I am not able to understand the purpose of concatenating the attention head vectors, when the researchers could have used some other method to fuse the vectors. I wanted to understand the intuition behind that. I got a very good conversation on stack exchange itself (Combine two feature vectors for a correct input of a neural network). Taking the guidance from the link I wanted to understand whether the attention head outputs are linearly separable and that's why the researchers decided to concatenate the vectors? Or is it something related to direct vector sum for subspaces (not sure), or there is any proof relating to Information Theory wherein somebody has shown that the information loss in lesser that way (I am speculating). Please guide me through the concept.

  • 1
    $\begingroup$ the attention head outputs are linearly separable this has nothing to do with concatenation... how would you combine them otherwise? you can choose to sum them, to weighted sum them, or to concatenate them, which is by far the most flexible, because the net will learn how to combine them $\endgroup$
    – Alberto
    Commented Aug 17, 2023 at 23:09
  • $\begingroup$ @AlbertoSinigaglia make an answer $\endgroup$
    – rjurney
    Commented Aug 18, 2023 at 5:00
  • $\begingroup$ Thanks @AlbertoSinigaglia, but why only concatenation, what's the benefit of that compared to the ways you have provided. $\endgroup$ Commented Aug 18, 2023 at 5:09
  • $\begingroup$ The other ones have some sort of biases, for example sum has the fact that the order is not relevant (1+2 = 2+1), however attention heads learn to attend to different concepts, so the invariance might not be a good idea $\endgroup$
    – Alberto
    Commented Aug 18, 2023 at 21:37


You must log in to answer this question.