Questions tagged [layer-normalization]

The tag has no usage guidance.

2 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
0 answers
634 views

Where should we place layer normalization in a transformer model?

In Attention Is All You Need paper: That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
Alexey Romanov's user avatar
1 vote
0 answers
29 views

How to choose proper normalization strategy for the activations?

I am reading a survey on various normalization techniques adopted in neural network architectures. The purpose of introducing normalization is understandable - to stabilize the training and avoid ...
spiridon_the_sun_rotator's user avatar