5
votes
What are the consequences of layer norm vs batch norm?
This is how I understand it.
Batch normalization is used to remove internal "covariate shift" (wich may be not the case) by normalizing the input for each hidden layer using the statistics ...
- 967
1
vote
Layer Norm in a ResNet MLP
Usually you insert the normalization layer (be it BatchNorm, LayerNorm or whatever) after the convolutional layer and before the activation layer, i.e. Conv + Norm + ReLU.
The original ResNet applies ...
- 915
1
vote
Layer Norm in a ResNet MLP
Two things:
Layer Norm wasn't invented before ResNet. ResNet still uses the regular Batch Norm.
The model to use Layer Norm is residual block is ConvNeXt. Based on this line, it applies LayerNorm ...
- 1,160
Only top scored, non community-wiki answers of a minimum length are eligible