# Position of layer normalization in transformer model

In Attention Is All You Need paper:

That is, the output of each sub-layer is $$LayerNorm(x+Sublayer(x))$$, where $$Sublayer(x)$$ is the function implemented by the sub-layer itself. We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized.

which makes the final formula $$LayerNorm(x+Dropout(Sublayer(x)))$$. However, in https://github.com/tensorflow/models/blob/0effd158ae1e6403c6048410f79b779bdf344d7d/official/transformer/model/transformer.py#L278-L288, I see

def __call__(self, x, *args, **kwargs):
# Preprocessing: apply layer normalization
y = self.layer_norm(x)

# Get layer output
y = self.layer(y, *args, **kwargs)

# Postprocessing: apply dropout and residual connection
if self.train:
y = tf.nn.dropout(y, 1 - self.postprocess_dropout)
return x + y


which ends up as $$x+Dropout(Sublayer(LayerNorm(x)))$$. Plus there are extra LayerNorms as final layers in both encoder and decoder stacks.

In a quick test, the performance of this model seems to be better than if I change back to the paper's order of operations. My question is: why? And could it be predicted in advance?

I note that Generating Long Sequences with Sparse Transformers uses the $$x+Dropout(Sublayer(LayerNorm(x)))$$ order, but doesn't discuss it, unlike the other changes it makes to Transformer.