2
$\begingroup$

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://arxiv.org/pdf/1506.07503.pdf. (it seems this attention is used in various forms of GNMT and text-to-speech sythesizers like tacotron-2 https://github.com/Rayhane-mamah/Tacotron-2).

Even after repeated readings of this paper and other articles about Attention-mechanism, I'm confused about the dimensions of the matrices used, as the paper doesn't seem to describe it. My understanding is:

  • If I have decoder hidden dim 1024, that means ${s_{i-1}}$ vector is 1024 length.

  • If I have encoder output dim 512, that means $h_{j}$ vector is 512 length.

  • If total inputs to encoder is 256, then number of $j$ can be from 1 to 256.

  • Since $W x S_{i-1}$ is a matrix multiply, it seems $cols(W)$ should match $rows(S_{i-1})$, but $rows(W)$ still remain undefined. Same seems true for matrices $V, U, w, b$.

This is page-3/4 from the paper above that describes Attention-layer:

enter image description here

I'm unsure how to make sense of this. Am I missing something, or can someone explain this?

What I don't understand is:

  • What is the dimension of previous alignment (denoted by $alpha_{i-1})$? Shouldn't it be total values of $j$ in $h_{j}$ (which is 256 and means total different encoder output states)?

  • What is the dimension of $f_{i,j}$ and convolution filter $F$? (the paper says $F$ belongs to $kxr$ shape but doesn't define $'r'$ anywhere). What is $'r'$ and what does $'k x r'$ mean here?

  • How are these unknown dimensions for matrices $'V, U, w, b'$ described above determined in this model?

Improve this question
$\endgroup$
5
  • $\begingroup$ I suggest you ask only one question per post. $\endgroup$ – nbro♦ Jun 4 '20 at 14:38
  • $\begingroup$ @nbro these are very related questions, and it seems answers to anyone is sort of dependent on others too. $\endgroup$ – Joe Black Jun 4 '20 at 15:29
  • $\begingroup$ i didn't receive or see any answer to these questions, so figured it'd be better if they are all together so if anyone who knows the answers to these can answer them in one go. $\endgroup$ – Joe Black Jun 4 '20 at 15:29
  • $\begingroup$ Only someone that has read the related papers and is an expert on the topic could answer such questions. But there aren't many experts on specific topics. That's why I am suggesting you split this post into simpler ones. $\endgroup$ – nbro♦ Jun 4 '20 at 15:47
  • $\begingroup$ Unfortunately none of the many tutorials explain the concept. Since even the original paper does not explain it, the best approach may just be to ask the authors. $\endgroup$ – Nav Jul 29 '20 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.