All Questions
Tagged with layer-normalization neural-networks
2 questions
2
votes
0
answers
697
views
Where should we place layer normalization in a transformer model?
In Attention Is All You Need paper:
That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
- 121
1
vote
0
answers
30
views
How to choose proper normalization strategy for the activations?
I am reading a survey on various normalization techniques adopted in neural network architectures.
The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
- 2,754