1
$\begingroup$

I actually went through the Keras' batch normalization tutorial and the description there puzzled me more.

Here are some facts about batch normalization that I read recently and want a deep explanation on it.

  1. If you froze all layers of neural networks to their random initialized weights, except for batch normalization layers, you can still get 83% accuracy on CIFAR10.

  2. When setting the trainable layer of batch normalization to false, it will run in inference mode and will not update its mean and variance statistics.

$\endgroup$
1
  • $\begingroup$ Which tutorial are you exactly referring to? Can you please edit your post to include the link to the tutorial that states those facts? $\endgroup$ – nbro Jul 12 '20 at 13:26
0
$\begingroup$

I'm not sure how just training the batch normalisation layer, you can get an accuracy of 83%. The batch normalisation layer parameters $\gamma^{(k)}$ and $\beta^{(k)}$, are used to scale and shift the normalised batch outputs. These parameters are learnt during the back-propagation step. For the $k$th layer, $$y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}$$ The scaling and shifting are done to ensure a non - linear activation being outputted by each layer. Because batch normalisation scales outputs between 0-1, some activation functions are linear within that range (Eg. $tahh$ and $sigmoid$)

Regarding the second fact however, the difference between training and inference mode is this. During training mode, the statistics of each batch norm layer $\mu_B$ and $\sigma^2_B$ is computed. This statistic is used in scaling and normalising the outputs of the batch norm layer to have 0 mean and unit variance. At the same time, the current batch statistic computed is also used to update the running mean and running variance of the population. $\mu_B[t]$ represents the current batch mean, $\sigma^2_B[t]$ represents the current batch variance, while $\mu'_B[t]$ and $\sigma'_B[t]$ represent the accumulated means and variance from the previous batches. The running mean and variance of the population is then updated as $$\mu'_B[t]=\mu'_B[t]× momentum+ \mu_B[t]×(1−momentum)$$ $$\sigma'^2_B[t]=\sigma'^2_B[t] × momentum + \sigma^2_B[t]×(1−momentum)$$

In inference mode, the batch normalisation uses the running mean and variance computed during training mode to scale and normalise inputs in the batch norm layer instead of the current batch mean and variance.

$\endgroup$
0
$\begingroup$

A batch normalisation layer is like a standard FC layer but instead of learning weights and bias', you learn means and variances and scale the whole layer by said means and variances.

Fact 1:

Because it behaves just like a normal layer, and can learn, with the right structure it will learn to get a high enough accuracy.

Fact 2

Disabling learning on a batch norm layer is just like disabling learning on any other layer. It will not updated any of its parameters, and in this case the parameters are the means and variances, and so these will not be updated.

$\endgroup$
0
$\begingroup$

The original paper by Sergey Ioffe and Christian Szegedy; https://arxiv.org/abs/1502.03167 "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" is very good. Make sure to go through the paper slowly and make annotations to truly understand it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.