According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is defined $$ w_i=\left({1\over N}{1\over P(i)}\right)^\beta $$ in which $1/N$ is the probability of drawing a sample uniformly from the buffer, and $P(i)$ is the probability of drawing a sample from PER.

I'm wondering if we have to do the same to the target of the actor when we apply PER to DDPG. That is, multiplying $-Q(s_i, \mu(s_i))$ by $w_i$, where $\mu$ is the outcome of the actor.

In my opinion, it is necessary. And I've done some experiments in the gym environment BipedalWalker-v2. The results, however, is quite confusing: I constantly get better performance when I do not apply importance sampling to the actor. Why would this be the case?



You must log in to answer this question.