I have to admit, that I still have problem digesting the HER algorithm proposed in the famous paper and despite the fact, that the idea behind it should be intuitive, I realize that it is not so easy to implement as expected. Even studying some of the implementations out there I cannot get a working version of my HER implementation.

Let's go straight to the point. The basic idea is: in case a given goal is not reached, then the trajectory collected in the episode should be replayed after changing the goal into a pseudo-goal. Just to keep it simple, let's take the case where the last reached state is the new goal. The data is then stored in the replay memory buffer $\mathbb{R}$:

$(s_{t} \parallel g, a_{t}, r_{t}, s_{t+1} \parallel g, d_{t}) \to \mathbb{R}$

Let's take a look into the algorithm:

enter image description here

In the original algorithm, the collection of the trajectory is done in the loop inside the orange box (in the next loop those collected data are store in the replay memory buffer - standard experience replay).

Now looking in the green box below, you can see that there is a loop going from $0$ to $T-1$ doing basically two things:

  1. Storing the trajectory into the replay buffer. Ok
  2. Sampling for additional goals for replay $G = \mathbb{S}(current episode)$

Given that, I want to take the last state as a pseudo-goal, does it make sense to "set" this goal at every step t from $0$ to $T-1$? Would it not generate $T-1$ identical trajectories, which basically fill up the replay memory buffer but do not contribute to create meaningful data to pass to a NN later on?

  • 1
    $\begingroup$ Instead of writing "a question about the algorithm", can you please just put your question in the title? Thanks $\endgroup$
    – nbro
    Commented Mar 24 at 2:39


You must log in to answer this question.