In the Transformer (adopted in BERT), we normalize the attention weights (dot product of keys and queries) using a softmax in the Scaled Dot-Product mechanism. It is unclear to me whether this normalization is performed on each row of the weight matrix or on the entire matrix. In the TensorFlow tutorial, it is performed on each row (axis=-1), and in the official TensorFlow code, it is performed on the entire matrix (axis=None). The paper doesn't give many details.
To me, both methods can make sense, but they have a strong impact. If on each row, then each value will have a roughly similar norm, because the sum of its weights is 1. If on the entire matrix, some values might be "extinguished" because all of its weights can be very close to zero.