Paper to cite about normalizing the inputs to a neural network [closed]

Question

We can read that the normalization of the data that is input to a neural network is important and is considered as a best practice, for example SO #1, SO #2 and many other places. It looks to me that there is no consensus on why it works, but there are a wide variety of possible reasons, including that the initialization of the weights is assuming normalized data, and that the gradient descent might become unstable if the input variables are scaled very differently (see here). I myself have also experienced in an application that applying Z-score normalization was essential in order to achieve good results. On the other hand, my understanding is that it is not theoretically proven that normalization will always allow to produce better results than without it, and it is just one of the techniques in the engineer's toolbox to try.

Now the question is, which paper to cite about that normalization of inputs is a best practice?

I tried to look for a good reference for it but I have not found a really good one. For example, I cannot find this in particular in the Goodfellow book. There is a lot about batch normalization (i.e., that is often applied in between layers, and the normalization parameters are learned), and some computer vision related normalizations, but not this. The Glorot paper does not mention normalization in this sense either (what they normalize is the initial weights). On the other hand, this technique is there in quite some blog posts and tutorials, e.g., this one.

Answer 1

I have found a related part in

C. C. Aggarwal. Neural Networks and Deep Learning: A Textbook. Springer International Publishing AG, 2018. isbn: 978-3-319-94462-3. doi: 10.1007/978-3-319-94463-0.

about "Additive preprocessing and mean-centering" and "Feature normalization" on page 127. It says:

It can be useful to mean-center the data in order to remove certain types of bias effects. Many algorithms in traditional machine learning (such as principal component analysis) also work with the assumption of mean-centered data. In such cases, a vector of column-wise means is subtracted from each data point. Mean-centering is often paired with standardization, which is discussed in the section of feature normalization.

Answer 2

The following paper is about time series and provides a good explanation.

Passalis, N., Kanniainen, J., Gabbouj, M. et al. Forecasting Financial Time Series Using Robust Deep Adaptive Input Normalization. J Sign Process Syst 93, 1235–1251 (2021). https://doi.org/10.1007/s11265-020-01624-0 PDF

One question that naturally arises from the aforementioned observations is why normalization seems to affect the performance of DL algorithms to this extent. To better understand this we need to consider two different phenomena that often arise. First, the employed normalization scheme significantly affects the generalization of the model on unknown data. For example, when the raw price is used as a feature, then a DL model operates on an fully unknown region of the input space when the price exceeds the range of price values used during the training. Methods such as sample-based standardization and instance normalization [14] can mitigate the effects of this to some extent, leading to improved generalization results. It is also worth noting that recent literature demonstrated that applying the appropriate input pre-processing Schemes, such as including the normalized first order differences, emphasizing on specific characteristics of the data, e.g., zero-crossing rate, can lead to significant improvements in forecasting precision and trading performance [18, 19], further highlighting the importance of appropriately pre-processing the input time series. The second reason is intrinsic to the nature of the gradient descent-based optimization that is employed for training DL models. Even though the input features carry the same amount of information regardless of the scaling applied to the individual features, the gradient descent’s updates that are used for training the model are affected by the scale of the input features. Therefore, features with large magnitude have a larger effect on the direction of the descent, often leading the model toward different local minima than a model for which all the input feature were normalized in the same scale. The normalization therefore acts as a way to ensure that each feature will have “equal” chances for affecting the optimization.