0
$\begingroup$

I'm trying to understand how to apply softmax in self attention layer. Let's say we have Query and Key matrix where the last row is for Paddings

enter image description here

In this case Z = Q*K_t would be something like this:

enter image description here

When we add the Mask we have:

enter image description here

Now the problem. When I apply the softmax function to each row, I get that the last one is not going to be populated with 0s. By defining S = softmax(Z) what we get is:

enter image description here

To get the output Y of the self attention layer, we have to multiply S by V (Values matrix):

enter image description here

And as a result, we have that y41 and y42 are both functions of Padding values. How to get rid of this problem?

$\endgroup$

0

You must log in to answer this question.

Browse other questions tagged .