In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible. Mostly a min-max scaler is used. Alternatively, there is also a standard scaler.
Why do you calculate the mean and standard deviation offline over the complete dataset before training? Couldn't this be calculated per batch or even per file? What is the disadvantage? Why doesn't anyone do this?
