1
$\begingroup$

I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My understanding is that on-policy is just a category created by humans meaning that "the default algorithm doesn't do off-policy optimization". But why does that prohibit adding off-policy elements?

$\endgroup$

2 Answers 2

2
$\begingroup$

On policy algorithms contain policy and/or value update calculations that assume data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at all without some kind of intervention.

As an example SARSA has the TD target (new estimate for $q(S_t,A_t)$) of:

$$R_{t+1} + \gamma q(S_{t+1},A_{t+1})$$

The main problem here is the value of $A_{t+1}$ (there's a couple of minor issues with $S_t, A_t$ as well, but those also apply to off policy and are why you usually want your behaviour policy to be close to the target policy).

In an experience replay table you will have historic data from when the policy was different. So the value of $A_{t+1}$ that might be stored may not be the one that the current policy would take. Learning from that action would skew the value table to learning about whatever policy has just been sampled, instead of more accurately improve knowledge of the current policy.

To fix this, you could resample the action choice, or even use an expected q value across all actions that SARSA could currently take. That fixes things, but actually it also changes your method to off-policy Expected SARSA . . .

Similar issues apply to using experience replay tables with other on policy methods. They become inaccurate if used naively, and if you fix the issues with that, you reinvent off-policy methods (when possible, not all on policy methods have an off policy equivalent)

$\endgroup$
2
$\begingroup$

In principle replay mechanisms as HER, PER, etc cannot be applied to on-policy algorithms, like SARSA and Policy gradient as stated by @Neil Slater.

Anyway, If you are willing to "adapt" your on-policy algorithm you can introduce importance sampling weights, $\rho$, that basically weight the actions $A_t$ coming from another policy (say $\pi_b$) compared to the current policy, $\pi$. For example the gradient of the RL objective would be weighted by: $$\rho = \frac{\pi(a\mid s)}{\pi_b(a\mid s)}$$

The ACER algorithm basically does this (see section 3) in order to work with a replay buffer, potentially improving sample-efficiency too. Moreover, HER is suitable for goal-based RL, meaning that the experience tuple also includes a goal $g\in G$, and so the policy should also account for $g$ and so the ratio $\rho$.

$\endgroup$
1
  • 1
    $\begingroup$ Worth noting that when using $A_t$ to $A_{t+n}$ to calculate a value, then you don't weight $A_t$ and if you are using a bootstrap from your target policy then you don't weight $A_{t+n}$ either. This is why you don't see importance sampling corrections in single step Q learning. $\endgroup$ Jul 10, 2023 at 5:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .