Questions tagged [layer-normalization]
The layer-normalization tag has no usage guidance.
4
questions with no upvoted or accepted answers
2
votes
1
answer
70
views
When should one not use Normalization Techniques in a deep learning models?
When should one not use normalization between layers like Batch Norm, Layer Norm, Instance Norm, and Group Norm in deep learning while training a DL model?
- 21
2
votes
0
answers
683
views
Where should we place layer normalization in a transformer model?
In Attention Is All You Need paper:
That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
- 121
1
vote
0
answers
29
views
How to choose proper normalization strategy for the activations?
I am reading a survey on various normalization techniques adopted in neural network architectures.
The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
- 2,724
0
votes
0
answers
21
views
In the Original Transformer, How exactly is layer normalization done?
I am assuming we are talking about the Transformer as written in "Attention is all you need" paper.
Let's say the in the normalization layer is m x n, where m is the context size and n is ...
- 233