Should the reward or the Q value be clipped for reinforcement learning

Question

When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-linear function approximators, such as neural networks, diverge aggressively. One way to help stabilize training is using reward clipping. Because the temporal difference Q-update is a bootstrapping method (i.e., uses a previously calculated value to compute the current prediction), a very large previously calculated Q-value can make the current reward relatively minuscule, thus making the current reward not impact the Q-update, eventually leading the agent to diverge.

To avoid this, we can try to avoid the large Q-value in the first place by clipping the reward between [1, -1].

But I have seen some other people say that instead of clipping the reward itself, we can instead clip the Q-value between an interval.

I was wondering which method is better for convergence, and under what assumptions / circumstances. I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

Answer 1

I'll start with the last question in your post:

I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.

I highly doubt there will be any such theoretical work. The problem is that these variants of clipping (clipping rewards and clipping $Q$ values) fundamentally modify the task / the original objective. Once you clip either of those things, you fundamentally change what your agent is trying to optimize for from what the original goal was. I don't think it's ever going to be possible to get any rigorous, theoretical proofs about which one would be better in general. You'd likely have to start out with some very strong assumptions on the reward structure in the original task to have any hope of proving anything here, but such strong assumptions make you lose generality.

---

Intuitively... I think reward clipping feels "safer" to me more often than clipping $Q$-values. Clipping $Q$-values seems more aggressive, it could be viewed as some combination of clipping rewards (if you clip $Q$-values to $[-1, 1]$, you're still at the very least also clipping all rewards to that range), but additionally also putting a constraint on how far in the future you're looking (in some sense). This whole argument is very handwavy though

I suppose, slightly less handwavy, you could say that reward clipping is definitely "better" (in the sense that you don't deviate as much from the original objective) in environments where rewards of similar magnitudes can be collected frequently. I struggle to really think of a situation where clipping $Q$-values would be a clear favorite based on intuition. I wouldn't be surprised if clipping $Q$-values may turn out to be better after empirical evaluation in some cases, but it's difficult to say where that would be. It will also very much depend on what range is chosen. Clipping rewards to a range of $[-1, 1]$ is very different from clipping $Q$-values to the same range.