# Why does batch norm standardize with sample mean/variance, when it also learns parameters to scale the mean/variance?

Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean $$\beta$$ and variance $$\gamma^2$$ to prevent things like activations from exploding or shifting during the learning process.

More specifically: $$\hat{x} = \displaystyle \frac{x - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\label{1}\tag{1}$$ $$BatchNorm_{\mu_t, \sigma_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$

where

• $$x$$ is the layer input of the layer
• $$\mu_t, \sigma_t$$ is the sample mean and standard deviation at time step $$t$$
• $$\epsilon$$ is a small constant, and
• $$\gamma$$ and $$\beta$$ are learnable parameters so that the output is not necessarily standardized to mean $$0$$ and variance $$1$$, but possibly to another mean and variance that may be better for the neural network.

My question is, why does BatchNorm first standardize the input $$x$$ to $$\hat{x}$$ before applying the learnable parameters $$\gamma$$ and $$\beta$$? Isn't this redundant? The parameters $$\gamma$$ and $$\beta$$ could learn to standardize the input themselves right?

In fact, as training progresses, $$\mu_t$$ and $$\sigma_t$$ becomes updated to new values $$\mu_{t+1}$$ and $$\sigma_{t+1}$$, so the learned parameters at that time step, $$\gamma_t$$ and $$\beta_t$$, no longer apply for time step $$t+1$$ since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of $$BatchNorm_{\mu_{t+1}, \sigma_{t+1}}(x)$$ to $$BatchNorm_{\mu_t, \sigma_t}(x)$$, which are two different functions right?

Why not just simply make it like this?

$$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$

This would simplify the calculation of the gradients, which would make learning faster to compute.

BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.

• That second formula reminds me of the re-parametrization trick (used e.g. in VAEs). Not sure if it's related or not, though, because I'm not currently familiar with the details of batch normalization. So, you could investigate this option.
– nbro
Jan 13, 2021 at 10:55