One of the reviewers of the HER paper (which was accepted as a NIPS conference paper) wrote

Overall, I'd say that it's not a huge/deep idea, but a very nice addition to the learning toolbox. When it is applicable, it seems like an idea that should be used. I worry that it is rarely applicable, though, without a tremendous amount of task insight. Consider the "kick the ball into the goal" example. If we want to speed up the learning of the task, we would likely need to use the idea to help guide the robot to the ball, to guide the robot's foot to make contact with the ball, and to guide the robot's aim to send the ball into the goal. At this level, it doesn't feel particularly different from (potential-based) reward shaping, which has been used in precisely this way.

Potential-based reward shaping (PBRS) consists in augmenting the original/natural reward function $r(s, a, s')$ with $F(s, s') = \gamma \phi(s') - \phi(s)$, where $s$ is the current state, $s'$ the next state after $s$, $\gamma$ the discount factor and $\phi$ is the so-called potential function (which can be anything: say what!?), i.e. the new shaped reward function is defined as

$$ r'(s, a, s') = r(s, a, s') + F(s, s') $$

How exactly is HER (which essentially turns every successful or unsuccessful trajectory into a successful trajectory, by making one of the states in that trajectory a fake goal state, then replays these unsuccessful trajectories) related to PBRS? Sure, both attempt to deal with sparse rewards (and I see how), and, intuitively, HER is a reward shaping technique. However, now, I am trying to understand if there is a formal attempt to mathematically connect these two approaches. More precisely, how would we formalize HER in terms of PBRS? Can we do that? Can we come up with a potential function that corresponds to what HER is doing? Is there any research on this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.