Questions tagged [layer-normalization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
14 votes
1 answer

What are the consequences of layer norm vs batch norm?

I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the ...
Alexander Soare's user avatar
2 votes
0 answers

Where should we place layer normalization in a transformer model?

In Attention Is All You Need paper: That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
Alexey Romanov's user avatar
1 vote
0 answers

How to choose proper normalization strategy for the activations?

I am reading a survey on various normalization techniques adopted in neural network architectures. The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
spiridon_the_sun_rotator's user avatar
0 votes
2 answers

Layer Norm in a ResNet MLP

Where do you insert layer norm in a residual block? After the addition or before the activation function (RELU in this case)?
postnubilaphoebus's user avatar