1
$\begingroup$

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards by an estimate of their standard deviation, which is updated over time.

However, I can't really find something about common practice for standardization/normalization of the constructed targets. Is it done on top of reward normalization? Do we just let the agent train for a while and then take an estimate of the standard deviation of the targets and scale with the estimate then, assuming we can't use any information of the environment's dynamics?

If this question is too broad to be answered in general, I'll quickly list the specifics of my environment.

  • Rewards are stochastic and approximately in the range [-0.2, 0) before scaling with outliers excluded
  • The agent receives a reward after each timestep
  • Given a state and action, the next state is deterministic
  • Very short time horizon -> 5 timesteps per episode
  • Action space is discrete and only consists of 11 actions
  • With reward scaling, the q-targets are approximately in the range [-3, 0] (I saved a list of targets and plotted a histogram a couple of times during training)
$\endgroup$
3
  • $\begingroup$ Is the NN set up for regression, with linear output in last layer, and MSE loss function? That would be default for most DQN. I don't have an answer for your main question, although my opinion would be that no scaling is required or likely to be useful with q targets in [-3, 0]. There are other radically different options, such as distributional return predictions, although I have only used them for very simple reward schemes (e.g. +1, 0, -1 at end of episode), no idea if they would help or hinder here. $\endgroup$ Apr 19 at 17:52
  • $\begingroup$ @NeilSlater thanks for the response and your view. The NN is indeed set up for regression with the Huber loss. $\endgroup$
    – Peter
    Apr 19 at 18:24
  • $\begingroup$ Not sure about DQN, but in actor-critic algorithms (e.g. ppo) the state is always normalized, sometimes the value function is normalized (in that case, the targets for the value function are normalized so the value function outputs a normalized prediction). Never heard of reward normalization. $\endgroup$
    – Taw
    Apr 23 at 22:33

0

You must log in to answer this question.

Browse other questions tagged .