There are 4 features in my state representation and they are in different ranges. So I'm thinking state normalization would reduce the bias on certain features. And also, in the problem I consider, there is a distribution shift in the inputs between train and test times. I think state normalization would help both issues.I found some suggestions in this question here
Can someone provide an example where this is done? It's a bit confusing to me.
Normalization, also known as scaling or standardization of input features, is essential not only in Reinforcement Learning but in machine learning in general. Normalization is widely accepted as a good practice, particularly when dealing with features that vary in scale and range. By doing so, you help ensure that the training process is both stable and efficient.
Here are a couple of main reasons:
Convergence Speed: Normalized inputs can accelerate the convergence of the algorithm by ensuring that the gradient descent steps are balanced across all dimensions. Without normalization, features with larger scales dominate the gradient updates, which can lead to slower convergence or divergence.
(source)
Equal Contribution: By scaling features to a similar range, you encourage the model to consider all features equally during learning. Otherwise, features with larger magnitudes may have a disproportionate influence on the distance metrics, loss calculation, and learned weights.
Better Learning Rate Generalization: When features are normalized, the learning rate parameter becomes more general and less dependent on the scale of the features. It's easier to find a suitable learning rate that works well across all features.
(source)
The normalization is done not only for the input data but also for intermediate layers. Features with large values can cause numerical instability, leading to issues such as exploding gradients in neural networks. Normalization keeps the values within a controlled range, contributing to the numerical stability of the training process. You can read more on it in Bach Normalization and Group Normalization papers. These papers provide theoretical information on normalization in general. We can consider the output of each hidden layer as an input to the next one. So some insights are also useful to understand normalization of the input data.
Depending on your data you may choose different approaches for normalizing your input data. For example, for images, it could be a simple mapping of pixel intensities [0, 255] to [0, 1] or [-1, 1] range.
Another common practice is to calculate the mean and standard deviation of your training set for normalization.
One more technique is min-max scaling, where each feature is scaled to a range [0, 1] or [-1, 1] by subtracting the minimum value and dividing by the range.
It's important to note that you should not recompute the mean and standard deviation for normalization during the testing phase or while the model is in production. Use the statistics computed from the training set to avoid any leakage and to ensure consistency between the training and testing normalization processes.
