5
$\begingroup$

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the infamous sparse reward problem. It is based on the intuition that human beings constantly try and learn something useful even from their past failed experiences.

I have almost completely understood the concept. But in the algorithm posited in the paper, I don't really understand how the optimization works. Once the fictitious trajectories are added, we have a state-goal-action dependency. This means our DQN should predict Q-Values based on an input state and the goal we're pursuing (The paper mentions how HER is extremely useful for multi-RL as well).

Does this mean I need to add another input feature (goal) to my DQN? An input state and an input goal, as two input features to my DQN, which is basically a CNN?

Because in the optimization step they have mentioned that we need to randomly sample trajectories from the replay buffer and use those for computing the gradients. It wouldn't make sense to compute the Q-Values without the goal now, because then we'd wind up with duplicate values.

Could someone help me understand how exactly does the optimization take place here?

I am training Atari's "Montezuma's Revenge" using a double DQN with Hindsight Experience Replay (HER).

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
0
$\begingroup$

Yes, you need to add goal state as an input otherwise you won't know what goal you are trying to pursue. It might take a lot of memory to run HER on ATARI games though, also defining final goal is not straightforward for such environments

Improve this answer
$\endgroup$
1
  • $\begingroup$ Maybe you should clarify why "defining final goal is not straightforward for such environments". What goal are you talking about? Are you talking about the fake goals that are needed to augment the ER? $\endgroup$
    – nbro
    Nov 20, 2020 at 18:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .