Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [layer-normalization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
44 views

When should one not use Normalization Techniques in a deep learning models?

When should one not use normalization between layers like Batch Norm, Layer Norm, Instance Norm, and Group Norm in deep learning while training a DL model?
Jivitesh's user avatar
0 votes
2 answers
1k views

Layer Norm in a ResNet MLP

Where do you insert layer norm in a residual block? After the addition or before the activation function (RELU in this case)?
postnubilaphoebus's user avatar
1 vote
0 answers
29 views

How to choose proper normalization strategy for the activations?

I am reading a survey on various normalization techniques adopted in neural network architectures. The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
spiridon_the_sun_rotator's user avatar
14 votes
1 answer
9k views

What are the consequences of layer norm vs batch norm?

I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the ...
Alexander Soare's user avatar
2 votes
0 answers
655 views

Where should we place layer normalization in a transformer model?

In Attention Is All You Need paper: That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
Alexey Romanov's user avatar