1
$\begingroup$

I am trying to understand the disentangled self-attention in DeBERTa paper, but I can't understand at which point of the paper they stop from introducing background knowledge about already known attention mechanisms and go on to discuss disentangled attention.

More specifically, in Section $3$ of DeBERTa, they state that each token is represented using $2$ vectors $\{H_i\}$, and $\{P_{i|j}\}$. Then they went onto introduce equation $(2)$: $$A_{i,j} = H_iH_j^T + H_iP_{j|i}^T + P_{i|j}H_{j}^T + P_{i|j}P_{j|i}^T$$

which they call calculating cross attention score. But then they also later go onto introduce another attention score in $(4)$ (pic below): Attention

What I don't understand is why there are $2$ different attention matrices $A, \tilde{A}$ each one having something to do with disentanglement (one disentangles word embedding, the other introduces disentangled weight matrices). Which one is the new novel method they introduced? Was $(2)$ just for some toy example giving? Also, they don't clarify how to produce vectors $H, P_{i|j}$ explicitly, could anyone explain how to process and input x to produce $H, P_{i|j}$

$\endgroup$

0

You must log in to answer this question.