I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5:
The estimation of the expected value with stochastic updates relies on those updates corresponding to the same distribution as its expectation. Prioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to (even if the policy and state distribution are fixed).
My understanding of this statement is that sampling non-uniformly from the replay memory is an issue.
So, my question is: Since we are working 1-step off-policy, why is it an issue? I thought that in an off-policy setting we don't care how transitions are sampled (at least in the 1-step case).
The one possibility for an issue that came to my mind is that in the particular case of PER, we are sampling transitions according to the errors and rewards, which does seem a little fishy.
A somewhat related question was asked here, but I don't think it answers my question.