3
$\begingroup$

I have asked this question a number of times, but I always get confusing answers to this, like "normalized data works better", "data lives in the same scale"

How can x-m/s make the scale of images the same? Please explain to me the maths. Also, take MNIST dataset for example & illustration.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

I answered a similar question earlier and here is a piece of my answer that i think covers your question:

Batch normalization's assistance to neural networks wasn't really understood for the longest time, initially it was thought to assist with internal covariate shift (hypothesized by the initial paper: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift) but lately has been tied to the optimization process (How Does Batch Normalization Help Optimization?).

This means, from an architectural perspective, it is difficult to correctly assume how it should be utilized or how it will effect your network, unless you really understand its impact on the loss landscape and how your optimization process will traverse it given your initialization (Note, by the way, a recent paper by Google showed that you can alleviate a lot of the benefits of batch normalization sheerly by understanding what issues it's resolving and attempting to mitigate them in the initialization process: Fixup Initialization).

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .