0
$\begingroup$

I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math surrounding how it is implemented. Specifically wouldn't batch normalization effectively change the output of each layer as the activations are now normalized? Also, any details on alpha and beta parameters would be greatly appreciated! Thank you!

$\endgroup$

1 Answer 1

1
$\begingroup$

I think it'd be helpful to refer to the batchnorm formula given in the PyTorch implementation.

In particular, given an input $x$, you would get the mean and variance ($\mathbf{E}[x]$ and $\text{Var}[x]$), and normalize it as such: $$ \hat{x} = \frac{x - \mathbf{E}[x]}{\sqrt{\text{Var}[x] + \epsilon}} $$ where $\epsilon$ is a small number to improve stability. There is also an additional affine transform applied in most implementations of batchnorm in standard libraries, so the output is really $\gamma \hat{x} + \beta$, where $\gamma$ and $\beta$ are the affine parameters (scale and shift respectively).

To answer your question (if I understood correctly), yes batchnorm would change the output of the previous layer, due to the normalization and affine transformation.

There are some details to consider:

  • In most implementations, the $\mathbf{E}[x]$ and $\text{Var}[x]$ are running mean estimates, meaning that there would be additional momentum hyperparameter to control how fast the estimates are updated as new inputs come in.
  • For batchnorm, you would normalize and apply the affine transformations to each output channel separately, meaning that each output channel would hold its own set of running mean estimates and affine parameters.
$\endgroup$
2
  • $\begingroup$ Thanks for the reply, just have a quick question. When you say moving mean do you mean in the mini-batch or for the final inference stage? Specifically are these moving means and variances used only during the inference stage? I am asking this because it seems like a lot of work to go through the training sample all over again with normalized outputs for each node, rather than just having a moving mean and average from the start of the mini-batch. $\endgroup$ Sep 6, 2022 at 6:28
  • $\begingroup$ @liyuzerihun The moving means are only updated during training, and updated per mini-batch when fed forward through the network. For inference, the standard practice is to "freeze" them (meaning that they do not get updated). $\endgroup$
    – PeaBrane
    Sep 6, 2022 at 6:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .