How to deal with small reward values

Question

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale up the rewards, say multiple by 100?

Answer 1

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might be based on average reward. You will generally know which is in use if you are building the environment. For instance, if you typically run episodes that are 1000 time steps long, then rewards in $[-0.01, 0.01]$ on each time step might be fine.

Neural networks are less sensitive to scale of outputs for regression problems than they are to scale of inputs. However it does affect things such as the scale of gradients for loss functions.

You could adjust for small expected returns by increasing the learning rate to compensate. But it should also be OK to re-scale the rewards so that the min and max returns are somewhere in e.g. $[-10, 10]$. The exact range is not a big deal, so you could pick a range where you are comfortable interpreting the values that you see quickly - multiplying by some factor of 10 makes it easier to map back to the original (and presumably meaningful) values in your head.

In the original Atari Game DQN paper, the authors normalised game scores for each of the games. This was mainly so that the same hyperparameters for the neural network worked in all the games (the agent was not tuned specially for each game). However, it may have also had some small benefit of keeping gradients within bounds for more efficient learning.