Batch norm is a normalizing layer that is shown to help deep networks learn faster and with higher generalization accuracy. It normalizes the activations of the previous layer to a mean $\beta$ and variance $\gamma^2$ to prevent things like activations from exploding or shifting during the learning process.
More specifically: $$\hat{x} = \displaystyle \frac{x - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}}\label{1}\tag{1}$$ $$ BatchNorm_{\mu_t, \sigma_t}(x) = \gamma \hat{x} + \beta \label{2}\tag{2}$$
where
- $x$ is the layer input of the layer
- $\mu_t, \sigma_t$ is the sample mean and standard deviation at time step $t$
- $\epsilon$ is a small constant, and
- $\gamma$ and $\beta$ are learnable parameters so that the output is not necessarily standardized to mean $0$ and variance $1$, but possibly to another mean and variance that may be better for the neural network.
My question is, why does BatchNorm first standardize the input $x$ to $\hat{x}$ before applying the learnable parameters $\gamma$ and $\beta$? Isn't this redundant? The parameters $\gamma$ and $\beta$ could learn to standardize the input themselves right?
In fact, as training progresses, $\mu_t$ and $\sigma_t$ becomes updated to new values $\mu_{t+1}$ and $\sigma_{t+1}$, so the learned parameters at that time step, $\gamma_t$ and $\beta_t$, no longer apply for time step $t+1$ since that involves a different standardization process with a different mean and variance. So by adding this standardization step, it may even hurt the convergence of the layer during learning, since it is adding the gradient of $BatchNorm_{\mu_{t+1}, \sigma_{t+1}}(x)$ to $BatchNorm_{\mu_t, \sigma_t}(x)$, which are two different functions right?
Why not just simply make it like this?
$$BatchNorm(x) = \gamma x + \beta \label{3}\tag{3}$$
This would simplify the calculation of the gradients, which would make learning faster to compute.
BatchNorm is one of the most successful developments of deep learning, so I know my intuition on these things is wrong -- I'm just curious as to what I am missing.
