Questions tagged [layer-normalization]
The layer-normalization tag has no usage guidance.
4
questions
14
votes
1
answer
7k
views
What are the consequences of layer norm vs batch norm?
I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the ...
- 1,319
2
votes
0
answers
581
views
Where should we place layer normalization in a transformer model?
In Attention Is All You Need paper:
That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
- 121
1
vote
0
answers
29
views
How to choose proper normalization strategy for the activations?
I am reading a survey on various normalization techniques adopted in neural network architectures.
The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
- 2,494
0
votes
2
answers
528
views
Layer Norm in a ResNet MLP
Where do you insert layer norm in a residual block?
After the addition or before the activation function (RELU in this case)?
