1
$\begingroup$

I want to understand the handshake between SGD (or mini-batch GD) and batch normalization. Below, an explanation quoted from this Medium article. However, I am confused about the denormalization by the SGD.

SGD ( Stochastic gradient descent) undoes this normalization if it’s a way for it to minimize the loss function.

Consequently, batch normalization adds two trainable parameters to each layer, so the normalized output is multiplied by a “standard deviation” parameter (gamma) and add a “mean” parameter (beta). In other words, batch normalization lets SGD do the denormalization by changing only these two weights for each activation, instead of losing the stability of the network by changing all the weights.

$\endgroup$
1
  • $\begingroup$ This Medium article! $\endgroup$
    – Arighna
    Mar 3 at 15:00

1 Answer 1

0
$\begingroup$

I think this phrasing is a bit misleading. If I understand this passage correctly, another way to put it would be:

Applying batch normalization distorts the true data distribution: An arbitrarily distributed batch of data is transformed into a distribution with mean = 0 and standard deviation = 1. This might not be beneficial for the downstream task, so two scalars beta and gamma are introduced that are trainable and can modify the output of the BatchNorm. This formula for BatchNorm makes this clear:

$\text{BatchNorm}({\mathcal{X}}_{b}) = \gamma \cdot \frac{(\mathcal{X}_{b} - \text{mean}(\mathcal{X}_{b}))}{\text{std}(\mathcal{X}_{b})} + \beta$

Without the gamma and beta, you would simply normalize the batch. But SGD can modify beta and gamma and thereby shift the distribution around according to what the optimization task demands. By acting on beta and gamma, the SGD can then somewhat 'counteract' this normalization.

$\endgroup$
2
  • $\begingroup$ Thank you for the detailed answer, it's crystal clear now! $\endgroup$
    – Arighna
    Mar 9 at 17:53
  • $\begingroup$ I'm glad it helped :) $\endgroup$
    – Chillston
    Mar 10 at 15:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .