In the Transformer (adopted in BERT), we normalize the attention weights (dot product of keys and queries) using a softmax in the Scaled Dot-Product mechanism. It is unclear to me whether this normalization is performed on each row of the weight matrix or on the entire matrix. In the TensorFlow tutorial, it is performed on each row (axis=-1), and in the official TensorFlow code, it is performed on the entire matrix (axis=None). The paper doesn't give many details.

To me, both methods can make sense, but they have a strong impact. If on each row, then each value will have a roughly similar norm, because the sum of its weights is 1. If on the entire matrix, some values might be "extinguished" because all of its weights can be very close to zero.

  • $\begingroup$ I might have been confused by the meaning of "axis=None" and "axis-1" which both actually select the last dimension. So the softmax is performed on the rows in both implementations. $\endgroup$ – Robin May 22 '19 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.