Following the DQN algorithm with experience replay:
We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.
Assume I have positive but changing rewards. Meaning, $r>0$.
Thus, since the rewards are positive, by calculating the loss, I notice that almost always $Q(s)< Q(s+1)+r$.
Therefore,the network learns to always increase the Q function , and eventually the Q function is higher in same states in later learning steps.
How can I stabilize the learning process?