Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean $\beta$ and variance $\gamma^2$ to prevent things like activations from exploding or shifting during the learning process.

More specifically: $$\hat{x} = \displaystyle \frac{x - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\label{1}\tag{1}$$ $$ BatchNorm_{\mu_t, \sigma_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$


  • $x$ is the layer input of the layer
  • $\mu_t, \sigma_t$ is the sample mean and standard deviation at time step $t$
  • $\epsilon$ is a small constant, and
  • $\gamma$ and $\beta$ are learnable parameters so that the output is not necessarily standardized to mean $0$ and variance $1$, but possibly to another mean and variance that may be better for the neural network.

My question is, why does BatchNorm first standardize the input $x$ to $\hat{x}$ before applying the learnable parameters $\gamma$ and $\beta$? Isn't this redundant? The parameters $\gamma$ and $\beta$ could learn to standardize the input themselves right?

In fact, as training progresses, $\mu_t$ and $\sigma_t$ becomes updated to new values $\mu_{t+1}$ and $\sigma_{t+1}$, so the learned parameters at that time step, $\gamma_t$ and $\beta_t$, no longer apply for time step $t+1$ since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of $BatchNorm_{\mu_{t+1}, \sigma_{t+1}}(x)$ to $BatchNorm_{\mu_t, \sigma_t}(x)$, which are two different functions right?

Why not just simply make it like this?

$$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$

This would simplify the calculation of the gradients, which would make learning faster to compute.

BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.

  • $\begingroup$ That second formula reminds me of the re-parametrization trick (used e.g. in VAEs). Not sure if it's related or not, though, because I'm not currently familiar with the details of batch normalization. So, you could investigate this option. $\endgroup$
    – nbro
    Jan 13, 2021 at 10:55

1 Answer 1


Theoretically, Yes and batchnorm can be omitted. Practically, No. Most of the intermediate features aren't normalized. For example, after a 2x2 Max pool of a normalized feature, the SD and mean would be vastly shifted to 0.8 and 1.2 (I don't remember the exact number) of the originals. It will take an immense amount of iterations for subsequent learnable parameters to absorb these scales and offsets. Before the absorption happens, those low SD feature map will be suppressed and lowering the overall network capacity. Other parameters are trained according the condition where some features are suppressed. The overall dynamics of unnormalized features are unpredictable. Batch Norm first normalize x, then the learnable gamma and beta are updated under the same scale of other normalized features.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .