Questions tagged [layer-normalization]
The layer-normalization tag has no usage guidance.
2
questions with no upvoted or accepted answers
2
votes
0
answers
581
views
Where should we place layer normalization in a transformer model?
In Attention Is All You Need paper:
That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
- 121
1
vote
0
answers
29
views
How to choose proper normalization strategy for the activations?
I am reading a survey on various normalization techniques adopted in neural network architectures.
The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
- 2,494