How does Hindsight Experience Replay learn from unsuccessful trajectories?

Question

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.

Ignoring HER for now, if in the case where the robotic arm reaches the goal correctly, then the value functions ($V$) and action-value functions ($Q$) that correspond to the trajectories that get to the goal quicker will increase. These high $Q$ and $V$ values are ultimately important for getting the optimal policy.

However, if you create 'fake' goals from unsuccessful trajectories - that would increase the $Q$ and $V$s of the environment that lead to getting the 'fake' goal. Those new $Q$ and $V$s would be unhelpful and possibly detrimental for the robotic arm to reach the real goal.

What am I misunderstanding?

Answer 1

Ignoring HER for now the $Q$ and $V$ functions operate on states and actions which are part of a Markov decision process which we call $M_0$.

Back to HER, the $Q$ and $V$ functions now take a goal as an additional parameter called the goal. We will denote individual goals $g_n$, the true goal $g_0$, and the set of all goals $G$. The set of goals is chosen such that every state matches at least one goal. We create a new MDP $M_1 = M_0 \times G$ (i.e. a larger MDP composed of multiple copies of $M_0$, all the states in each being tagged with a goal). The reward is either +1 or 0 depending on whether or not the goal and the component from $M_0$ match in some predefined sense. In HER trajectories are collected from the subset of $M_1$ where $g = g_0$ and added to the replay buffer. When training the $Q$ and $V$ functions we don't only use the original trajectories: we create new ones by substituting new values of $g$, which we do so strategically so as to include some trajectories with a positive reward.

Things to note:

1. HER doesn't assign rewards to states or trajectories in $M_0$: the reward function used is only defined for $M_1$
2. The performance of HER depends on $Q$ and $V$ both being models with some ability to extrapolate to unseen data points; such as neural networks, support vector machines, etc. It would not provide any benefit if applied to value tables.