I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along with the PER samples to downplay each sample's loss contribution according to its probability of selection. I've observed that typically these sample weightings vary from $~0.1$ to $<0.01$, as the buffer gradually fills up (4.8M samples).
When using this compensation, the growth of the maximal Q value per episode stalls prematurely compared to a non-weight-compensated regime. I presume that this is because the size of the back-propagation updates is being greatly and increasingly diminished by the sample weights.
To correct for this I've tried taking the beta-adjusted maximum weight as reported by the PER (the same buffer-wide value by which the batch is normalized) and multiplying the base learning rate by it, thereby adjusting the optimizer after each batch selection.
My question is two-fold:
Is this the correct interpretation of what's going on?
Is it standard practice to compensate for sample weighting in this way?
Although it seems to be working in keeping the Q growth alive whilst taming the loss, I've not been able to find any information on this and haven't found any implementations that compensate in this way so have a major doubt about the mathematical validity of it.