1
$\begingroup$

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.

Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions (V) and action-value functions (Q) that correspond to the trajectories that get to the goal quicker will increase. These high Q and V values are ultimately important for getting the optimal policy.

However if you create 'fake' goals from unsuccessful trajectories - that would increase the Q and Vs of the environment that lead to getting the 'fake' goal. Those new Q and Vs would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.

What am I misunderstanding?

|improve this question
$\endgroup$
  • $\begingroup$ Can you quote the part of the paper where they claim that these "fake" goals are created? In which sense they are "fake"? I think that they do no reward these fake goals in the same way they reward the actual goal. $\endgroup$ – nbro Nov 16 '18 at 11:14
  • $\begingroup$ I don't think they used the term 'fake'. I think they said 'virtual'. How do you they treat these fake goals differently. I'm super confused even though I have read a few articles about HER. Would greatly appreciate if you could add an answer if you have a good insight to this algorithm. $\endgroup$ – piccolo Nov 17 '18 at 13:58
  • $\begingroup$ As I said, as far as I understood, these intermediate goals are not rewarded in the same way as the actual goal, so the agent will ultimately prefer the actual goal (which is rewarded more), I think. $\endgroup$ – nbro Nov 17 '18 at 13:59
0
$\begingroup$

The functions comprising your agent have an extra parameter representing the current goal. So V(s) in standard Q learning becomes V(g,s) and Q(s,a) becomes Q(g,s,a). If we call the real goal g0 then we might collect training data from episodes sampled using Q(g0,s,a), etc. But during replay we don't just use g0 but also substitute g1, g2, g3 etc which should include potential goals that the agent has actually achieved.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.