Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://arxiv.org/pdf/1506.07503.pdf. (it seems this attention is used in various forms of GNMT and text-to-speech sythesizers like tacotron-2 https://github.com/Rayhane-mamah/Tacotron-2).
Even after repeated readings of this paper and other articles about Attention-mechanism, I'm confused about the dimensions of the matrices used, as the paper doesn't seem to describe it. My understanding is:
If I have decoder hidden dim 1024, that means ${s_{i-1}}$ vector is 1024 length.
If I have encoder output dim 512, that means $h_{j}$ vector is 512 length.
If total inputs to encoder is 256, then number of $j$ can be from 1 to 256.
Since $W x S_{i-1}$ is a matrix multiply, it seems $cols(W)$ should match $rows(S_{i-1})$, but $rows(W)$ still remain undefined. Same seems true for matrices $V, U, w, b$.
This is page-3/4 from the paper above that describes Attention-layer:
I'm unsure how to make sense of this. Am I missing something, or can someone explain this?
What I don't understand is:
What is the dimension of previous alignment (denoted by $alpha_{i-1})$? Shouldn't it be total values of $j$ in $h_{j}$ (which is 256 and means total different encoder output states)?
What is the dimension of $f_{i,j}$ and convolution filter $F$? (the paper says $F$ belongs to $kxr$ shape but doesn't define $'r'$ anywhere). What is $'r'$ and what does $'k x r'$ mean here?
How are these unknown dimensions for matrices $'V, U, w, b'$ described above determined in this model?