1
$\begingroup$

In most implementations of neural networks the features are scaled to make the optimization of the loss function as stable as possible. Mostly a min-max scaler is used. Alternatively, there is also a standard scaler.

Why do you calculate the mean and standard deviation offline over the complete dataset before training? Couldn't this be calculated per batch or even per file? What is the disadvantage? Why doesn't anyone do this?

$\endgroup$
1
  • 2
    $\begingroup$ Just to clarify, you're wondering why we couldn't take a batch from the data, compute the mean and standard deviation of each feature over that batch, and then scale using that (so the means from each batch could potentially be different)? $\endgroup$
    – htl
    Apr 9 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.