When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-linear function approximators, such as neural networks, diverge aggressively. One way to help stabilize training is using reward clipping. Because the temporal difference Q-update is a bootstrapping method (i.e., uses a previously calculated value to compute the current prediction), a very large previously calculated Q-value can make the current reward relatively minuscule, thus making the current reward not impact the Q-update, eventually leading the agent to diverge.

To avoid this, we can try to avoid the large Q-value in the first place by clipping the reward between [1, -1].

But I have seen some other people say that instead of clipping the reward itself, we can instead clip the Q-value between an interval.

I was wondering which method is better for convergence, and under what assumptions / circumstances. I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

  • 1
    You could clip the TD Error, which has the benefit of not modifying the objective, just the size of update step that you can take at one time. That might in fact be related to what you are referring to - perhaps give a reference/link to the reward or q value clipping document(s) that you read, in order to clarify that. – Neil Slater Oct 16 at 19:19
up vote 2 down vote accepted

I'll start with the last question in your post:

I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I highly doubt there will be any such theoretical work. The problem is that these variants of clipping (clipping rewards and clipping $Q$ values) fundamentally modify the task / the original objective. Once you clip either of those things, you fundamentally change what your agent is trying to optimize for from what the original goal was. I don't think it's ever going to be possible to get any rigorous, theoretical proofs about which one would be better in general. You'd likely have to start out with some very strong assumptions on the reward structure in the original task to have any hope of proving anything here, but such strong assumptions make you lose generality.


Intuitively... I think reward clipping feels "safer" to me more often than clipping $Q$-values. Clipping $Q$-values seems more aggressive, it could be viewed as some combination of clipping rewards (if you clip $Q$-values to $[-1, 1]$, you're still at the very least also clipping all rewards to that range), but additionally also putting a constraint on how far in the future you're looking (in some sense). This whole argument is very handwavy though

I suppose, slightly less handwavy, you could say that reward clipping is definitely "better" (in the sense that you don't deviate as much from the original objective) in environments where rewards of similar magnitudes can be collected frequently. I struggle to really think of a situation where clipping $Q$-values would be a clear favorite based on intuition. I wouldn't be surprised if clipping $Q$-values may turn out to be better after empirical evaluation in some cases, but it's difficult to say where that would be. It will also very much depend on what range is chosen. Clipping rewards to a range of $[-1, 1]$ is very different from clipping $Q$-values to the same range.

  • Thanks for the answer Dennis. Yes, the clipping Q values would indeed affect the "prediction horizon" for RL. That is a great point! I will probably try both and see which one is more interesting, but for Q clipping, I will try to clip it at a higher interval to not affect the prediction horizon. – Rui Nian Oct 16 at 20:35

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.