2
$\begingroup$

In Attention Is All You Need paper:

That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to the output of each sub-layer, before it is added to the sub-layer input and normalized.

which makes the final formula $LayerNorm(x+Dropout(Sublayer(x)))$. However, in https://github.com/tensorflow/models/blob/0effd158ae1e6403c6048410f79b779bdf344d7d/official/transformer/model/transformer.py#L278-L288, I see

def __call__(self, x, *args, **kwargs):
  # Preprocessing: apply layer normalization
  y = self.layer_norm(x)

  # Get layer output
  y = self.layer(y, *args, **kwargs)

  # Postprocessing: apply dropout and residual connection
  if self.train:
    y = tf.nn.dropout(y, 1 - self.postprocess_dropout)
  return x + y

which ends up as $x+Dropout(Sublayer(LayerNorm(x)))$. Plus there are extra LayerNorms as final layers in both encoder and decoder stacks.

In a quick test, the performance of this model seems to be better than if I change back to the paper's order of operations. My question is: why? And could it be predicted in advance?

I note that Generating Long Sequences with Sparse Transformers uses the $x+Dropout(Sublayer(LayerNorm(x)))$ order, but doesn't discuss it, unlike the other changes it makes to Transformer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.