4 votes

What are the consequences of layer norm vs batch norm?

This is how I understand it. Batch normalization is used to remove internal "covariate shift" (wich may be not the case) by normalizing the input for each hidden layer using the statistics ...
Aray Karjauv's user avatar
1 vote

Layer Norm in a ResNet MLP

Usually you insert the normalization layer (be it BatchNorm, LayerNorm or whatever) after the convolutional layer and before the activation layer, i.e. Conv + Norm + ReLU. The original ResNet applies ...
pi-tau's user avatar
  • 612
1 vote

Layer Norm in a ResNet MLP

Two things: Layer Norm wasn't invented before ResNet. ResNet still uses the regular Batch Norm. The model to use Layer Norm is residual block is ConvNeXt. Based on this line, it applies LayerNorm ...
Minh-Long Luu's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible