According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is defined $$ w_i=\left({1\over N}{1\over P(i)}\right)^\beta $$ in which $1/N$ is the probability of drawing a sample uniformly from the buffer, and $P(i)$ is the probability of drawing a sample from PER.

I'm wondering if we have to do the same to the target of the actor when we apply PER to DDPG. That is, multiplying $-Q(s_i, \mu(s_i))$ by $w_i$, where $\mu$ is the outcome of the actor.

In my opinion, it is necessary. And I've done some experiments in the gym environment BipedalWalker-v2. The results, however, is quite confusing: I constantly get better performance when I do not apply importance sampling to the actor. Why would this be the case?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.