Self Attention Diagram

In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?

  • $\begingroup$ Please, edit your post to format the formulas with mathjax. $\endgroup$
    – nbro
    Dec 28, 2023 at 1:07
  • $\begingroup$ Also, please, put your specific question in the title. Thanks. $\endgroup$
    – nbro
    Dec 28, 2023 at 12:25


You must log in to answer this question.

Browse other questions tagged .