Which generalization of standard deviation to use for multidimensional input normalization

Question

For machine learning tasks, it's common to normalize input data by subtracting the mean $\mu$ and dividing by the standard deviation $\sigma$ of the dataset:

$$\hat{x_i} = \frac{x_i - \mu}{\sigma}$$

—where the standard deviation is computed independently for each feature.

Suppose instead we compute another multidimensional generalization of the standard deviation—the matrix square root $\Sigma$ of the covariance matrix:

$$\Sigma\Sigma^T = \frac{1}{n}\sum_i^n (x_i - \mu)(x_i - \mu)^T$$

(Since covariance matrix $\Sigma\Sigma^T$ is positive semidefinite, it has a unique positive semidefinite square root $\Sigma$.)

And then normalize the samples by

$$\hat{x_i} = \Sigma^{-1}(x_i - \mu)$$

In a sense this is a "better" normalization—it will not only map the values to nice ranges, but also decorrelate different features.

However, I have never seen this done. Is there some non-obvious drawback to this approach I'm not seeing, aside from being more computationally intensive? Is there any research on what effect this would have on model training?

Answer 1

This idea is sometimes applied in computer vision, under the name of Whitening Transform, or ZCA sphering transform. The name whitening comes from signal processing, since removing correlation from a signal makes it look like white noise.

In images this transformation is applied to remove contrast and enhance edges, as you can see in the example below (taken from this paper, not open access unfortunately, but the authors made a github repo)

I'm not aware either about application of covariance matrix in domains other than computer vision for preprocessing, but I can image few points that make this approach unpractical:

• Feature selection: it's easier to select features that are already uncorrelated (checking trough chi squared test or similar) rather than turning them into uncorrelated ones. And this approach has the advantage of reducing training features and therefore speed up training and reduce memory requirements.
• Impracticality, for thousands of training features computing the covariance matrix become basically impossible, not just inefficient. This problem becomes relevant even with less features if we want to train with batches.
• Generalization It seems to me your also suggesting the possibility to compute a unique covariance matrix from a set of training data. This approach would force us apply the same covariance matrix on every out of training data on which we want to apply our model, meaning that after applying that covariance matrix our data will probably not be truly covariance free. This is a problem common to standardization as well (the mean and variance computed on the training data always differ from the real mean and std of the population, but it's easy to compute mean and std of millions of training instances, so usually those estimation are quite robust).