In Attention Is All You Need paper:

That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized.

which makes the final formula $LayerNorm(x+Dropout(Sublayer(x)))$. However, in https://github.com/tensorflow/models/blob/0effd158ae1e6403c6048410f79b779bdf344d7d/official/transformer/model/transformer.py#L278-L288, I see

def __call__(self, x, *args, **kwargs):
  # Preprocessing: apply layer normalization
  y = self.layer_norm(x)

  # Get layer output
  y = self.layer(y, *args, **kwargs)

  # Postprocessing: apply dropout and residual connection
  if self.train:
    y = tf.nn.dropout(y, 1 - self.postprocess_dropout)
  return x + y

which ends up as $x+Dropout(Sublayer(LayerNorm(x)))$. Plus there are extra LayerNorms as final layers in both encoder and decoder stacks.

In a quick test, the performance of this model seems to be better than if I change back to the paper's order of operations. My question is: why? And could it be predicted in advance?

I note that Generating Long Sequences with Sparse Transformers uses the $x+Dropout(Sublayer(LayerNorm(x)))$ order, but doesn't discuss it, unlike the other changes it makes to Transformer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.