0
$\begingroup$

In the Transformer (adapted in BERT), we normalize the attention weights (dot product of keys and queries) using a softmax in the Scaled Dot-Product mechanism. It is unclear to me whether this normalization is performed on each row of the weight matrix or on the entire matrix. In the tensorflow tutorial, it is performed on each row (axis=-1), and in the official tensorflow code, it is performed on the entire matrix (axis=None). The paper doesn't give much details.

To me, both methods can make sens but they have a strong impact. If on each row, then each value will have a roughly similar norm because its weights some to one. If on the entire matrix, some values might be "extinguished" because all of its weights can be very close to zero.

$\endgroup$
  • $\begingroup$ I might have been confused by the meaning of "axis=None" and "axis-1" which both actually select the last dimension. So the softmax is performed on the rows in both implementations. $\endgroup$ – debzsud May 22 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.