In the Transformer (adapted in BERT), we normalize the attention weights (dot product of keys and queries) using a softmax in the Scaled Dot-Product mechanism. It is unclear to me whether this normalization is performed on each row of the weight matrix or on the entire matrix. In the tensorflow tutorial, it is performed on each row (axis=-1), and in the official tensorflow code, it is performed on the entire matrix (axis=None). The paper doesn't give much details.
To me, both methods can make sens but they have a strong impact. If on each row, then each value will have a roughly similar norm because its weights some to one. If on the entire matrix, some values might be "extinguished" because all of its weights can be very close to zero.