I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards by an estimate of their standard deviation, which is updated over time.
However, I can't really find something about common practice for standardization/normalization of the constructed targets. Is it done on top of reward normalization? Do we just let the agent train for a while and then take an estimate of the standard deviation of the targets and scale with the estimate then, assuming we can't use any information of the environment's dynamics?
If this question is too broad to be answered in general, I'll quickly list the specifics of my environment.
- Rewards are stochastic and approximately in the range [-0.2, 0) before scaling with outliers excluded
- The agent receives a reward after each timestep
- Given a state and action, the next state is deterministic
- Very short time horizon -> 5 timesteps per episode
- Action space is discrete and only consists of 11 actions
- With reward scaling, the q-targets are approximately in the range [-3, 0] (I saved a list of targets and plotted a histogram a couple of times during training)