When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-linear function approximators, such as neural networks, diverge aggressively. One way to help stabilize training is using reward clipping. Because the temporal difference Q-update is a bootstrapping method (i.e., uses a previously calculated value to compute the current prediction), a very large previously calculated Q-value can make the current reward relatively minuscule, thus making the current reward not impact the Q-update, eventually leading the agent to diverge.
To avoid this, we can try to avoid the large Q-value in the first place by clipping the reward between [1, -1].
But I have seen some other people say that instead of clipping the reward itself, we can instead clip the Q-value between an interval.
I was wondering which method is better for convergence, and under what assumptions / circumstances. I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.