Following the DQN algorithm with experience replay:

dqn algorithm

We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.

Assume I have positive but changing rewards. Meaning, $r>0$.

Thus, since the rewards are positive, by calculating the loss, I notice that almost always $Q(s)< Q(s+1)+r$.

Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps.

How can I stabilize the learning process?

  • $\begingroup$ Could you clarify whether you are working on an episodic or continuous problem? $\endgroup$ – Neil Slater Apr 24 at 18:50
  1. You can use discount factor gamma less then one.

  2. You can use finite time horizon - only for states which are no farther away then T time steps reward propagate back

  3. You can use sum of rewords averaged over time for Q

All of those are legitimate approaches.

  • $\begingroup$ @mimirror2image 1. My discounr factor is 0.9. This increasing process happens even with discount factor 0.1. 2. How to calculate it? The network should calculate it. 3.Sum of rewards will still be > 0, which returns to the same procedure of increasing q function $\endgroup$ – BestR Apr 24 at 15:23

Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps

If your value function keeps increasing in later steps that means that the network is still learning those Q-values, you shouldn't necessarily prevent that. Your Q-values won't increase forever even if the rewards are always positive. You basically have a regression problem here and when the value of $Q(s,a)$ becomes very close to the predicted value of $r+Q(s',a)$ value of $Q(s,a)$ will stop increasing by itself.

  • $\begingroup$ Yes, but than the rewards are "swollen" in the Q function std. The Q function should'nt be more than 5*max_reward. $\endgroup$ – BestR Apr 24 at 15:20
  • 1
    $\begingroup$ sorry, I didn't quite understand, are you saying that Q-value can't be more than 5 times maximum possible reward ? What makes you think that. $\endgroup$ – Brale_ Apr 24 at 15:41
  • $\begingroup$ Im saying it should not be. Otherwise, the rewards will be swollen in the q function standard deviation $\endgroup$ – BestR Apr 24 at 16:04
  • $\begingroup$ @BestR: there is no basis in theory for your statement about this limit to the action value - its value is supposed to be the discounted sum of future rewards, and that can be any arbitrary amount times individual rewards, depending on how you have set up the environment. Please could you explain more where this constraint that you want has come from, as it may help clarify your question? $\endgroup$ – Neil Slater Apr 24 at 18:53
  • $\begingroup$ I'm still a bit confused, why exactly are you calculating std of Q-values, it's not part of the DQN algorithm or are you doing that for your own statistics ? Also, what do you mean by rewards get "swollen" in Q-value std, why exactly are you using rewards to calculate std of Q-values. Are you maybe changing your rewards based on the value of std for Q-values ? $\endgroup$ – Brale_ Apr 25 at 7:35

I changed the rewards to be negative and positive by substructing the mean reward.

It seems to improve the Q function boundries.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.