in the original paper the following scheme of the self-attention appears: https://arxiv.org/pdf/1805.08318.pdf enter image description here

In a later overview: https://arxiv.org/pdf/1906.01529.pdf

this scheme appears: enter image description here referring the original paper.

My understanding more correlates with the second paper scheme, as: enter image description here Where there is two dot-product operations and three hidden parametric matrices: $$W_k, W_v, W_q$$ which corresponds to $W_f, W_g, W_h$ without $W_v$ as it in the original paper explanation, which is as following:

enter image description here

Is this a mistake in the original paper ?

  • 1
    $\begingroup$ I don't understand where the mistake is. Can you please clarify where the mistake is? $\endgroup$ – nbro Mar 29 at 10:36
  • $\begingroup$ I think I was confused by the notation of v as a vector function, I thought it is a vector and then it didn't work out. So eventually there is non. $\endgroup$ – Ilya.K. Mar 29 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.